[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:07:54 dist-test-slave-p1lc krb5kdc[8553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:07:54 dist-test-slave-p1lc krb5kdc[8553](info): set up 2 sockets
Sep 28 13:07:54 dist-test-slave-p1lc krb5kdc[8553](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 13:07:55.529830 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:07:55 dist-test-slave-p1lc krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:07:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:07:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:07:55.656723 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:34443
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:34443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:07:55.742671 8566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:07:55.742930 8566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:07:55.742972 8566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:07:55.744197 8566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:07:55.744266 8566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:07:55.744290 8566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:07:55.744321 8566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:07:55.744347 8566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:07:55.745954 8566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:34443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:34443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:07:55.746244 8566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:07:55.746484 8566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:07:55.752276 8572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:07:55.752344 8566 server_base.cc:1034] running on GCE node
W20260928 13:07:55.752276 8574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:55.752315 8571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:07:55.752842 8566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:07:55.753196 8566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:07:55.754388 8566 hybrid_clock.cc:648] HybridClock initialized: now 1790600875754367 us; error 41 us; skew 500 ppm
Sep 28 13:07:55 dist-test-slave-p1lc krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:07:55.757274 8566 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:07:55.757906 8566 webserver.cc:460] Webserver started at http://127.8.88.190:41595/ using document root <none> and password file <none>
I20260928 13:07:55.758242 8566 fs_manager.cc:359] Metadata directory not provided
I20260928 13:07:55.758332 8566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:07:55.758479 8566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:07:55.759624 8566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "ba2fd7202e224a85b62fd736806b7999"
format_stamp: "Formatted at 2026-09-28 13:07:55 on dist-test-slave-p1lc"
server_key: "2795d208f0b8b0270d4a21c109c90195"
server_key_iv: "1fccb7cddb5f883344f674c0dd7a7e59"
server_key_version: "encryptionkey@0"
I20260928 13:07:55.760000 8566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "ba2fd7202e224a85b62fd736806b7999"
format_stamp: "Formatted at 2026-09-28 13:07:55 on dist-test-slave-p1lc"
server_key: "2795d208f0b8b0270d4a21c109c90195"
server_key_iv: "1fccb7cddb5f883344f674c0dd7a7e59"
server_key_version: "encryptionkey@0"
I20260928 13:07:55.761560 8566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:07:55.762392 8581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:55.762590 8566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:07:55.762699 8566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "ba2fd7202e224a85b62fd736806b7999"
format_stamp: "Formatted at 2026-09-28 13:07:55 on dist-test-slave-p1lc"
server_key: "2795d208f0b8b0270d4a21c109c90195"
server_key_iv: "1fccb7cddb5f883344f674c0dd7a7e59"
server_key_version: "encryptionkey@0"
I20260928 13:07:55.762774 8566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:07:55.773886 8566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:07:55.776266 8566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:07:55.776454 8566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:07:55.780699 8566 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:34443
I20260928 13:07:55.780735 8633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:34443 every 8 connection(s)
I20260928 13:07:55.781191 8566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:07:55.781991 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 8566
I20260928 13:07:55.782212 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:07:55.782397 8634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:07:55.783063 8546 external_mini_cluster.cc:1428] Setting key 0dbff822da929a0d27600beb23e32bbf
I20260928 13:07:55.785669 8634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999: Bootstrap starting.
I20260928 13:07:55.786424 8634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999: Neither blocks nor log segments found. Creating new log.
I20260928 13:07:55.786767 8634 log.cc:824] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999: Log is configured to *not* fsync() on all Append() calls
I20260928 13:07:55.787612 8634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999: No bootstrap required, opened a new log
I20260928 13:07:55.789819 8634 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2fd7202e224a85b62fd736806b7999" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 34443 } }
I20260928 13:07:55.789992 8634 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:07:55.790033 8634 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba2fd7202e224a85b62fd736806b7999, State: Initialized, Role: FOLLOWER
I20260928 13:07:55.790169 8634 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2fd7202e224a85b62fd736806b7999" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 34443 } }
I20260928 13:07:55.790241 8634 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:07:55.790269 8634 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:07:55.790325 8634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:07:55.790956 8634 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2fd7202e224a85b62fd736806b7999" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 34443 } }
I20260928 13:07:55.791044 8634 leader_election.cc:302] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba2fd7202e224a85b62fd736806b7999; no voters:
I20260928 13:07:55.791249 8634 leader_election.cc:288] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:07:55.791304 8639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:07:55.791483 8639 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [term 1 LEADER]: Becoming Leader. State: Replica: ba2fd7202e224a85b62fd736806b7999, State: Running, Role: LEADER
I20260928 13:07:55.791532 8634 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:07:55.791692 8639 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2fd7202e224a85b62fd736806b7999" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 34443 } }
I20260928 13:07:55.792097 8641 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba2fd7202e224a85b62fd736806b7999. Latest consensus state: current_term: 1 leader_uuid: "ba2fd7202e224a85b62fd736806b7999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2fd7202e224a85b62fd736806b7999" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 34443 } } }
I20260928 13:07:55.792244 8641 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [sys.catalog]: This master's current role is: LEADER
I20260928 13:07:55.792517 8647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:07:55.792677 8640 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba2fd7202e224a85b62fd736806b7999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba2fd7202e224a85b62fd736806b7999" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 34443 } } }
I20260928 13:07:55.792742 8640 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999 [sys.catalog]: This master's current role is: LEADER
I20260928 13:07:55.793061 8647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 13:07:55 dist-test-slave-p1lc krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:07:55.798120 8647 catalog_manager.cc:1347] Generated new cluster ID: 777495d68c4f4094a5b5b311eb4bf81a
I20260928 13:07:55.798180 8647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:07:55.801631 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:55.784094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.8.88.190:34443)
0928 13:07:55.784303 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:55.784316 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:55.784567 (+ 251us) server_negotiation.cc:407] Connection header received
0928 13:07:55.786945 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:55.786955 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:55.787018 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:55.787165 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:55.788800 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:55.790287 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:55.791660 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:55.792276 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:55.792391 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:55.797307 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:55.797311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:55.797311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:55.797323 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:55.799463 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:55.799993 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:55.799996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:55.799997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:55.800110 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:55.800403 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:55.800404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:55.800404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:55.800694 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:55.800840 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:55.801270 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 13:07:55.801424 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260928 13:07:55.831493 8647 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:07:55.832060 8647 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:07:55.841379 8647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba2fd7202e224a85b62fd736806b7999: Generated new TSK 0
I20260928 13:07:55.841593 8647 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:07:55.877725 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:34443
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:07:55.967165 8660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:07:55.967378 8660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:07:55.967403 8660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:07:55.968562 8660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:07:55.968602 8660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:07:55.968629 8660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:07:55.970168 8660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:34443
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:07:55.970373 8660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:07:55.970562 8660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:07:55.975907 8665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:55.975916 8666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:55.975921 8668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:07:55.976338 8660 server_base.cc:1034] running on GCE node
I20260928 13:07:55.976558 8660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:07:55.976837 8660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:07:55.978085 8660 hybrid_clock.cc:648] HybridClock initialized: now 1790600875978027 us; error 80 us; skew 500 ppm
Sep 28 13:07:55 dist-test-slave-p1lc krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:07:55.981202 8660 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:07:55.981802 8660 webserver.cc:460] Webserver started at http://127.8.88.129:38473/ using document root <none> and password file <none>
I20260928 13:07:55.982062 8660 fs_manager.cc:359] Metadata directory not provided
I20260928 13:07:55.982133 8660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:07:55.982290 8660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:07:55.983392 8660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "8f2034be8be84cccab709960007b29b3"
format_stamp: "Formatted at 2026-09-28 13:07:55 on dist-test-slave-p1lc"
server_key: "3ee1a5e1d41a157a73385464f5cc4933"
server_key_iv: "9e1219c315d30e8e3a0432d4747c7e64"
server_key_version: "encryptionkey@0"
I20260928 13:07:55.983780 8660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "8f2034be8be84cccab709960007b29b3"
format_stamp: "Formatted at 2026-09-28 13:07:55 on dist-test-slave-p1lc"
server_key: "3ee1a5e1d41a157a73385464f5cc4933"
server_key_iv: "9e1219c315d30e8e3a0432d4747c7e64"
server_key_version: "encryptionkey@0"
I20260928 13:07:55.985417 8660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:07:55.986336 8675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:55.986529 8660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:07:55.986639 8660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "8f2034be8be84cccab709960007b29b3"
format_stamp: "Formatted at 2026-09-28 13:07:55 on dist-test-slave-p1lc"
server_key: "3ee1a5e1d41a157a73385464f5cc4933"
server_key_iv: "9e1219c315d30e8e3a0432d4747c7e64"
server_key_version: "encryptionkey@0"
I20260928 13:07:55.986723 8660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:07:56.000052 8660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:07:56.002339 8660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:07:56.002521 8660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:07:56.002789 8660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:07:56.003123 8660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:07:56.003180 8660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:56.003216 8660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:07:56.003301 8660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:56.009482 8660 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:43583
I20260928 13:07:56.009522 8788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:43583 every 8 connection(s)
I20260928 13:07:56.009867 8660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:07:56.014256 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 8660
I20260928 13:07:56.014362 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:07:56.014529 8546 external_mini_cluster.cc:1428] Setting key 14cb8fcbfe303f5059127e4edfe66319
Sep 28 13:07:56 dist-test-slave-p1lc krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:07:56.021205 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:56.010730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:50919 (local address 127.8.88.190:34443)
0928 13:07:56.010832 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:56.010834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:56.011025 (+ 191us) server_negotiation.cc:407] Connection header received
0928 13:07:56.011603 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:56.011604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:56.011617 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:56.011665 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:56.012681 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.013269 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.014445 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.014753 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.014840 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:56.017265 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:56.017267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:56.017268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:56.017278 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:56.019015 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:56.019551 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:56.019552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:56.019552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:56.019633 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:56.020339 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:56.020340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:56.020341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:56.020487 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:56.020555 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:56.021015 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 13:07:56.021127 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 13:07:56.021516 8791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:56.010637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:34443 (local address 127.8.88.129:50919)
0928 13:07:56.010864 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 13:07:56.010918 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 13:07:56.011486 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:07:56.011738 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:07:56.011743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:07:56.012364 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:07:56.012566 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:56.012570 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.013366 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.013367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:56.014325 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:56.014328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.014919 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.014920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:56.015070 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:56.015667 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:07:56.015676 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:07:56.017171 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:07:56.019104 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:56.019112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:56.019114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:56.019467 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:56.019726 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:56.019727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:56.019728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:56.019901 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:56.020575 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:07:56.020585 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:07:56.020814 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 13:07:56.020988 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 13:07:56.021230 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260928 13:07:56.022182 8789 heartbeater.cc:347] Connected to a master server at 127.8.88.190:34443
I20260928 13:07:56.022310 8789 heartbeater.cc:464] Registering TS with master...
I20260928 13:07:56.022500 8789 heartbeater.cc:511] Master 127.8.88.190:34443 requested a full tablet report, sending...
I20260928 13:07:56.023101 8598 ts_manager.cc:194] Registered new tserver with Master: 8f2034be8be84cccab709960007b29b3 (127.8.88.129:43583)
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
I20260928 13:07:56.023782 8598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:50919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:07:56.046200 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:34443
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:07:56.128651 8794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:07:56.128943 8794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:07:56.128986 8794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:07:56.130420 8794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:07:56.130525 8794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:07:56.130580 8794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:07:56.132265 8794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:34443
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:07:56.132567 8794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:07:56.132827 8794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:07:56.138187 8800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:56.138273 8799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:56.138366 8802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:07:56.138432 8794 server_base.cc:1034] running on GCE node
I20260928 13:07:56.138814 8794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:07:56.139073 8794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:07:56.140398 8794 hybrid_clock.cc:648] HybridClock initialized: now 1790600876140378 us; error 29 us; skew 500 ppm
Sep 28 13:07:56 dist-test-slave-p1lc krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600876, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:07:56.143186 8794 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:07:56.143671 8794 webserver.cc:460] Webserver started at http://127.8.88.130:42761/ using document root <none> and password file <none>
I20260928 13:07:56.143929 8794 fs_manager.cc:359] Metadata directory not provided
I20260928 13:07:56.144002 8794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:07:56.144164 8794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:07:56.145314 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "92d30d20060c4c2a9792e8f0627cb160"
format_stamp: "Formatted at 2026-09-28 13:07:56 on dist-test-slave-p1lc"
server_key: "37b73aa4450d0ce21e7fc5a66307d7be"
server_key_iv: "995f3f5dbe192eb23acf0dc1289e4ad1"
server_key_version: "encryptionkey@0"
I20260928 13:07:56.145685 8794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "92d30d20060c4c2a9792e8f0627cb160"
format_stamp: "Formatted at 2026-09-28 13:07:56 on dist-test-slave-p1lc"
server_key: "37b73aa4450d0ce21e7fc5a66307d7be"
server_key_iv: "995f3f5dbe192eb23acf0dc1289e4ad1"
server_key_version: "encryptionkey@0"
I20260928 13:07:56.147269 8794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:07:56.148219 8809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:56.148453 8794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:07:56.148546 8794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "92d30d20060c4c2a9792e8f0627cb160"
format_stamp: "Formatted at 2026-09-28 13:07:56 on dist-test-slave-p1lc"
server_key: "37b73aa4450d0ce21e7fc5a66307d7be"
server_key_iv: "995f3f5dbe192eb23acf0dc1289e4ad1"
server_key_version: "encryptionkey@0"
I20260928 13:07:56.148602 8794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:07:56.168217 8794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:07:56.170836 8794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:07:56.170964 8794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:07:56.171186 8794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:07:56.171573 8794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:07:56.171646 8794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:56.171693 8794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:07:56.171717 8794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:56.177951 8794 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:34613
I20260928 13:07:56.177991 8922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:34613 every 8 connection(s)
I20260928 13:07:56.178345 8794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:07:56.181911 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 8794
I20260928 13:07:56.182011 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:07:56.182282 8546 external_mini_cluster.cc:1428] Setting key 1d9d108e6f2726c83455ef8c492dfd94
Sep 28 13:07:56 dist-test-slave-p1lc krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600876, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:07:56.190232 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:56.179190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:48591 (local address 127.8.88.190:34443)
0928 13:07:56.179254 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:56.179256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:56.179487 (+ 231us) server_negotiation.cc:407] Connection header received
0928 13:07:56.179985 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:56.179986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:56.179997 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:56.180065 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:56.181128 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.181651 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.183149 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.183348 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.183411 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:56.186245 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:56.186248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:56.186248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:56.186256 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:56.188215 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:56.188712 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:56.188713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:56.188714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:56.188787 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:56.189267 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:56.189269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:56.189269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:56.189436 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:56.189519 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:56.189999 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 13:07:56.190104 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 13:07:56.190356 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:56.179180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:34443 (local address 127.8.88.130:48591)
0928 13:07:56.179353 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 13:07:56.179399 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 13:07:56.179886 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:07:56.180126 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:07:56.180129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:07:56.180738 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:07:56.181026 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:56.181031 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.181768 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.181769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:56.183005 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:56.183009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.183444 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.183445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:56.183654 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:56.184282 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:07:56.184293 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:07:56.186131 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:07:56.188333 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:56.188351 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:56.188353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:56.188622 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:56.188958 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:56.188959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:56.188965 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:56.189171 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:56.189538 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:07:56.189545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:07:56.189801 (+ 256us) client_negotiation.cc:769] Sending connection context
0928 13:07:56.189908 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 13:07:56.190078 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260928 13:07:56.190999 8923 heartbeater.cc:347] Connected to a master server at 127.8.88.190:34443
I20260928 13:07:56.191097 8923 heartbeater.cc:464] Registering TS with master...
I20260928 13:07:56.191322 8923 heartbeater.cc:511] Master 127.8.88.190:34443 requested a full tablet report, sending...
I20260928 13:07:56.191828 8598 ts_manager.cc:194] Registered new tserver with Master: 92d30d20060c4c2a9792e8f0627cb160 (127.8.88.130:34613)
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
I20260928 13:07:56.192461 8598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:48591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:07:56.213919 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:34443
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:07:56.299916 8928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:07:56.300169 8928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:07:56.300209 8928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:07:56.301479 8928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:07:56.301548 8928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:07:56.301594 8928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:07:56.303351 8928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:34443
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:07:56.303630 8928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:07:56.303875 8928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:07:56.309252 8936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:56.309355 8934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:56.309398 8933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:07:56.309646 8928 server_base.cc:1034] running on GCE node
I20260928 13:07:56.309839 8928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:07:56.310103 8928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:07:56.311259 8928 hybrid_clock.cc:648] HybridClock initialized: now 1790600876311244 us; error 38 us; skew 500 ppm
Sep 28 13:07:56 dist-test-slave-p1lc krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600876, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:07:56.314106 8928 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:07:56.314497 8928 webserver.cc:460] Webserver started at http://127.8.88.131:33457/ using document root <none> and password file <none>
I20260928 13:07:56.314682 8928 fs_manager.cc:359] Metadata directory not provided
I20260928 13:07:56.314724 8928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:07:56.314812 8928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:07:56.315834 8928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "4861efa2640a4174a40d19f264dbe2cc"
format_stamp: "Formatted at 2026-09-28 13:07:56 on dist-test-slave-p1lc"
server_key: "e363c9847a1a74fa732bfade42d0cfe5"
server_key_iv: "d94cb8747ceefcdd3f66173d5e34685a"
server_key_version: "encryptionkey@0"
I20260928 13:07:56.316179 8928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "4861efa2640a4174a40d19f264dbe2cc"
format_stamp: "Formatted at 2026-09-28 13:07:56 on dist-test-slave-p1lc"
server_key: "e363c9847a1a74fa732bfade42d0cfe5"
server_key_iv: "d94cb8747ceefcdd3f66173d5e34685a"
server_key_version: "encryptionkey@0"
I20260928 13:07:56.317571 8928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:07:56.318339 8943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:56.318517 8928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:07:56.318621 8928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "4861efa2640a4174a40d19f264dbe2cc"
format_stamp: "Formatted at 2026-09-28 13:07:56 on dist-test-slave-p1lc"
server_key: "e363c9847a1a74fa732bfade42d0cfe5"
server_key_iv: "d94cb8747ceefcdd3f66173d5e34685a"
server_key_version: "encryptionkey@0"
I20260928 13:07:56.318718 8928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:07:56.337976 8928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:07:56.340400 8928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:07:56.340569 8928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:07:56.340826 8928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:07:56.341164 8928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:07:56.341220 8928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:56.341254 8928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:07:56.341320 8928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:56.347487 8928 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:42317
I20260928 13:07:56.347527 9056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:42317 every 8 connection(s)
I20260928 13:07:56.347846 8928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:07:56.350077 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 8928
I20260928 13:07:56.350267 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:07:56.350430 8546 external_mini_cluster.cc:1428] Setting key c949e3ae50305ed05901d0f468fae5cf
Sep 28 13:07:56 dist-test-slave-p1lc krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600876, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:07:56.358552 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:56.348697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:51325 (local address 127.8.88.190:34443)
0928 13:07:56.348764 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:56.348766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:56.349012 (+ 246us) server_negotiation.cc:407] Connection header received
0928 13:07:56.349987 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:56.349989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:56.350004 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:56.350086 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:56.351197 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.351903 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.353145 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.353339 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.353406 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:56.355676 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:56.355678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:56.355678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:56.355685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:56.357126 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:56.357611 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:56.357612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:56.357613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:56.357673 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:56.357936 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:56.357937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:56.357937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:56.358053 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:56.358125 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:56.358412 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 13:07:56.358484 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 13:07:56.358701 9059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:56.348694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:34443 (local address 127.8.88.131:51325)
0928 13:07:56.348836 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 13:07:56.348910 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 13:07:56.349865 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:07:56.350211 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:07:56.350215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:07:56.350891 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:07:56.351107 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:56.351111 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.351986 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.351987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:56.353063 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:56.353066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.353422 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.353422 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:56.353533 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:56.354091 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:07:56.354100 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:07:56.355553 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:07:56.357234 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:56.357241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:56.357243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:56.357493 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:56.357743 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:56.357744 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:56.357744 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:56.357874 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:56.358120 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:07:56.358124 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:07:56.358243 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 13:07:56.358344 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 13:07:56.358475 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":85,"threads_started":1}
I20260928 13:07:56.359277 9057 heartbeater.cc:347] Connected to a master server at 127.8.88.190:34443
I20260928 13:07:56.359390 9057 heartbeater.cc:464] Registering TS with master...
I20260928 13:07:56.359570 9057 heartbeater.cc:511] Master 127.8.88.190:34443 requested a full tablet report, sending...
I20260928 13:07:56.359998 8598 ts_manager.cc:194] Registered new tserver with Master: 4861efa2640a4174a40d19f264dbe2cc (127.8.88.131:42317)
I20260928 13:07:56.360394 8598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:51325
I20260928 13:07:56.362622 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 13:07:56 dist-test-slave-p1lc krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 13:07:56 dist-test-slave-p1lc krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600876, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.88.129@KRBTEST.COM
I20260928 13:07:56.387449 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:56.378411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.8.88.129:43583)
0928 13:07:56.378581 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:56.378596 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:56.378611 (+ 15us) server_negotiation.cc:407] Connection header received
0928 13:07:56.378650 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:56.378652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:56.378668 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:56.378740 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:56.379378 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.380176 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.381101 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.381458 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.381522 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:56.384250 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:56.384273 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:56.384274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:56.384282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:56.386054 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:56.386491 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:56.386495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:56.386496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:56.386565 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:56.386826 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:56.386827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:56.386827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:56.386998 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:56.387167 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:56.387217 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 13:07:56.387313 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
W20260928 13:07:56.387817 8703 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55536
Sep 28 13:07:56 dist-test-slave-p1lc krb5kdc[8553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 13:07:56 dist-test-slave-p1lc krb5kdc[8553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.129@KRBTEST.COM
I20260928 13:07:56.412400 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:56.405129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55538 (local address 127.8.88.129:43583)
0928 13:07:56.405198 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:56.405200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:56.405329 (+ 129us) server_negotiation.cc:407] Connection header received
0928 13:07:56.405377 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:56.405378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:56.405387 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:56.405448 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:56.406052 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.406555 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.407370 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:56.407607 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:56.407678 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:56.410085 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:56.410086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:56.410087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:56.410094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:56.411436 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:56.411756 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:56.411757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:56.411757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:56.411807 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:56.411973 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:56.411973 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:56.411974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:56.412064 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:56.412193 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:56.412266 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 13:07:56.412332 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 13:07:56.413050 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 8660
I20260928 13:07:56.417958 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 8794
I20260928 13:07:56.424023 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 8928
I20260928 13:07:56.428574 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 8566
2026-09-28T13:07:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1976 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:07:56 dist-test-slave-p1lc krb5kdc[9071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:07:56 dist-test-slave-p1lc krb5kdc[9071](info): set up 2 sockets
Sep 28 13:07:56 dist-test-slave-p1lc krb5kdc[9071](info): commencing operation
krb5kdc: starting...
W20260928 13:07:57.497098 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:07:57 dist-test-slave-p1lc krb5kdc[9071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:07:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:07:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:07:57.628980 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46767
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:07:57.710543 9084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:07:57.710777 9084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:07:57.710801 9084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:07:57.712000 9084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:07:57.712049 9084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:07:57.712064 9084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:07:57.712076 9084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:07:57.712088 9084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:07:57.713709 9084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:07:57.713979 9084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:07:57.714244 9084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:07:57.719614 9092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:07:57.719642 9084 server_base.cc:1034] running on GCE node
W20260928 13:07:57.719606 9089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:57.719601 9090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:07:57.720010 9084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:07:57.720391 9084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:07:57.721580 9084 hybrid_clock.cc:648] HybridClock initialized: now 1790600877721549 us; error 63 us; skew 500 ppm
Sep 28 13:07:57 dist-test-slave-p1lc krb5kdc[9071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600877, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:07:57.724298 9084 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:07:57.724817 9084 webserver.cc:460] Webserver started at http://127.8.88.190:33063/ using document root <none> and password file <none>
I20260928 13:07:57.725092 9084 fs_manager.cc:359] Metadata directory not provided
I20260928 13:07:57.725167 9084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:07:57.725315 9084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:07:57.726404 9084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "8fba2c900fb94db2a8631421e48fdf62"
format_stamp: "Formatted at 2026-09-28 13:07:57 on dist-test-slave-p1lc"
server_key: "e2370fc012e288c315b6579886c2db2c"
server_key_iv: "3e4e137999d3eb2e9d35122f5ebfccbf"
server_key_version: "encryptionkey@0"
I20260928 13:07:57.726780 9084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "8fba2c900fb94db2a8631421e48fdf62"
format_stamp: "Formatted at 2026-09-28 13:07:57 on dist-test-slave-p1lc"
server_key: "e2370fc012e288c315b6579886c2db2c"
server_key_iv: "3e4e137999d3eb2e9d35122f5ebfccbf"
server_key_version: "encryptionkey@0"
I20260928 13:07:57.728394 9084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:07:57.729494 9099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:57.729760 9084 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:07:57.729890 9084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "8fba2c900fb94db2a8631421e48fdf62"
format_stamp: "Formatted at 2026-09-28 13:07:57 on dist-test-slave-p1lc"
server_key: "e2370fc012e288c315b6579886c2db2c"
server_key_iv: "3e4e137999d3eb2e9d35122f5ebfccbf"
server_key_version: "encryptionkey@0"
I20260928 13:07:57.729978 9084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:07:57.744642 9084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:07:57.747032 9084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:07:57.747218 9084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:07:57.751451 9084 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:46767
I20260928 13:07:57.751489 9151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:46767 every 8 connection(s)
I20260928 13:07:57.751854 9084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:07:57.752777 9152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:07:57.754092 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 9084
I20260928 13:07:57.754163 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:07:57.754344 8546 external_mini_cluster.cc:1428] Setting key c81d25ea38c8a2e93f9c7db2ace8f106
I20260928 13:07:57.755772 9152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62: Bootstrap starting.
I20260928 13:07:57.756422 9152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62: Neither blocks nor log segments found. Creating new log.
I20260928 13:07:57.756750 9152 log.cc:824] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62: Log is configured to *not* fsync() on all Append() calls
I20260928 13:07:57.757620 9152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62: No bootstrap required, opened a new log
I20260928 13:07:57.760002 9152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fba2c900fb94db2a8631421e48fdf62" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46767 } }
I20260928 13:07:57.760180 9152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:07:57.760228 9152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fba2c900fb94db2a8631421e48fdf62, State: Initialized, Role: FOLLOWER
I20260928 13:07:57.760358 9152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fba2c900fb94db2a8631421e48fdf62" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46767 } }
I20260928 13:07:57.760462 9152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:07:57.760504 9152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:07:57.760557 9152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:07:57.761210 9152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fba2c900fb94db2a8631421e48fdf62" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46767 } }
I20260928 13:07:57.761343 9152 leader_election.cc:302] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fba2c900fb94db2a8631421e48fdf62; no voters:
I20260928 13:07:57.761541 9152 leader_election.cc:288] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:07:57.761626 9157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:07:57.761795 9157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [term 1 LEADER]: Becoming Leader. State: Replica: 8fba2c900fb94db2a8631421e48fdf62, State: Running, Role: LEADER
I20260928 13:07:57.761828 9152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:07:57.761888 9157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fba2c900fb94db2a8631421e48fdf62" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46767 } }
Sep 28 13:07:57 dist-test-slave-p1lc krb5kdc[9071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:07:57.762195 9158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fba2c900fb94db2a8631421e48fdf62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fba2c900fb94db2a8631421e48fdf62" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46767 } } }
I20260928 13:07:57.762274 9158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [sys.catalog]: This master's current role is: LEADER
I20260928 13:07:57.762528 9159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fba2c900fb94db2a8631421e48fdf62. Latest consensus state: current_term: 1 leader_uuid: "8fba2c900fb94db2a8631421e48fdf62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fba2c900fb94db2a8631421e48fdf62" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46767 } } }
I20260928 13:07:57.762840 9163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:07:57.763273 9159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62 [sys.catalog]: This master's current role is: LEADER
I20260928 13:07:57.763770 9163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:07:57.765549 9163 catalog_manager.cc:1347] Generated new cluster ID: bbe7b533603744fe95fa5267d117d71b
I20260928 13:07:57.765604 9163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:07:57.766907 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:57.754948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38054 (local address 127.8.88.190:46767)
0928 13:07:57.755150 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:57.755159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:57.755198 (+ 39us) server_negotiation.cc:407] Connection header received
0928 13:07:57.755756 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:57.755767 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:57.755831 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:57.755981 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:57.756612 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:57.758077 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:57.758894 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:57.759648 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:57.759782 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:57.762814 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:57.762819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:57.762819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:57.762833 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:57.765199 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:57.765692 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:57.765695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:57.765696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:57.765805 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:57.766025 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:57.766026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:57.766027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:57.766292 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:57.766455 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:57.766567 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 13:07:57.766725 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260928 13:07:57.774281 9163 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:07:57.775022 9163 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:07:57.779198 9163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fba2c900fb94db2a8631421e48fdf62: Generated new TSK 0
I20260928 13:07:57.779440 9163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:07:57.812639 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46767
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:07:57.897217 9178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:07:57.897442 9178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:07:57.897468 9178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:07:57.898685 9178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:07:57.898732 9178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:07:57.898761 9178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:07:57.900342 9178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:07:57.900596 9178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:07:57.900851 9178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:07:57.905905 9184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:57.905933 9186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:57.905983 9183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:07:57.906270 9178 server_base.cc:1034] running on GCE node
I20260928 13:07:57.906567 9178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:07:57.906827 9178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:07:57.907990 9178 hybrid_clock.cc:648] HybridClock initialized: now 1790600877907964 us; error 42 us; skew 500 ppm
Sep 28 13:07:57 dist-test-slave-p1lc krb5kdc[9071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600877, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:07:57.910912 9178 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:07:57.911434 9178 webserver.cc:460] Webserver started at http://127.8.88.129:46139/ using document root <none> and password file <none>
I20260928 13:07:57.911630 9178 fs_manager.cc:359] Metadata directory not provided
I20260928 13:07:57.911672 9178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:07:57.911765 9178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:07:57.912806 9178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4"
format_stamp: "Formatted at 2026-09-28 13:07:57 on dist-test-slave-p1lc"
server_key: "e2cb310b6686d800e4d730ccc2b2fb6f"
server_key_iv: "d6f11e32de4e50e39f9c36fe82e971ce"
server_key_version: "encryptionkey@0"
I20260928 13:07:57.913209 9178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4"
format_stamp: "Formatted at 2026-09-28 13:07:57 on dist-test-slave-p1lc"
server_key: "e2cb310b6686d800e4d730ccc2b2fb6f"
server_key_iv: "d6f11e32de4e50e39f9c36fe82e971ce"
server_key_version: "encryptionkey@0"
I20260928 13:07:57.914556 9178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:07:57.915320 9193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:57.915589 9178 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:07:57.915697 9178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4"
format_stamp: "Formatted at 2026-09-28 13:07:57 on dist-test-slave-p1lc"
server_key: "e2cb310b6686d800e4d730ccc2b2fb6f"
server_key_iv: "d6f11e32de4e50e39f9c36fe82e971ce"
server_key_version: "encryptionkey@0"
I20260928 13:07:57.915773 9178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:07:57.933279 9178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:07:57.935700 9178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:07:57.935827 9178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:07:57.936035 9178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:07:57.936379 9178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:07:57.936415 9178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:57.936439 9178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:07:57.936488 9178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:57.942667 9178 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:39579
I20260928 13:07:57.942714 9306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:39579 every 8 connection(s)
I20260928 13:07:57.943046 9178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:07:57.947965 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 9178
I20260928 13:07:57.948133 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:07:57.948309 8546 external_mini_cluster.cc:1428] Setting key c8e11b214cacf22acefd1ae6e898d145
Sep 28 13:07:57 dist-test-slave-p1lc krb5kdc[9071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600877, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:07:57.953125 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:57.943759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:33699 (local address 127.8.88.190:46767)
0928 13:07:57.943804 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:57.943806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:57.944003 (+ 197us) server_negotiation.cc:407] Connection header received
0928 13:07:57.945064 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:57.945065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:57.945078 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:57.945135 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:57.945805 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:57.946350 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:57.947389 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:57.947617 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:57.947704 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:57.950095 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:57.950097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:57.950097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:57.950105 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:57.951650 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:57.952094 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:57.952095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:57.952095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:57.952157 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:57.952421 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:57.952422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:57.952423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:57.952546 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:57.952620 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:57.952943 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 13:07:57.953032 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 13:07:57.953333 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:57.943732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46767 (local address 127.8.88.129:33699)
0928 13:07:57.943879 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 13:07:57.943916 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:07:57.944947 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:07:57.945212 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:07:57.945218 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:07:57.945495 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:07:57.945693 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:57.945697 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:57.946464 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:57.946465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:57.947292 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:57.947294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:57.947716 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:57.947717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:57.947849 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:57.948472 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:07:57.948483 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:07:57.949985 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:07:57.951754 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:57.951762 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:57.951764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:57.952000 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:57.952226 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:57.952227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:57.952228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:57.952351 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:57.952622 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:07:57.952629 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:07:57.952773 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 13:07:57.952905 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 13:07:57.953076 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260928 13:07:57.953929 9307 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46767
I20260928 13:07:57.954077 9307 heartbeater.cc:464] Registering TS with master...
I20260928 13:07:57.954335 9307 heartbeater.cc:511] Master 127.8.88.190:46767 requested a full tablet report, sending...
I20260928 13:07:57.955022 9116 ts_manager.cc:194] Registered new tserver with Master: ddbe6fb6e05d40c4b56ff679b24cfde4 (127.8.88.129:39579)
I20260928 13:07:57.956003 9116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:33699
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:07:57.981012 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46767
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:07:58.066531 9312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:07:58.066823 9312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:07:58.066872 9312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:07:58.068306 9312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:07:58.068394 9312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:07:58.068444 9312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:07:58.070259 9312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:07:58.070591 9312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:07:58.070861 9312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:07:58.076534 9318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:07:58.076591 9312 server_base.cc:1034] running on GCE node
W20260928 13:07:58.076601 9320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:58.076534 9317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:07:58.077080 9312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:07:58.077363 9312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:07:58.078548 9312 hybrid_clock.cc:648] HybridClock initialized: now 1790600878078528 us; error 43 us; skew 500 ppm
Sep 28 13:07:58 dist-test-slave-p1lc krb5kdc[9071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600878, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:07:58.081555 9312 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:07:58.082003 9312 webserver.cc:460] Webserver started at http://127.8.88.130:40999/ using document root <none> and password file <none>
I20260928 13:07:58.082224 9312 fs_manager.cc:359] Metadata directory not provided
I20260928 13:07:58.082269 9312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:07:58.082366 9312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:07:58.083463 9312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "0b92dcf9321f4885b76ceaf863e7ec46"
format_stamp: "Formatted at 2026-09-28 13:07:58 on dist-test-slave-p1lc"
server_key: "76040b959e75b10d5fcee33a72ffe678"
server_key_iv: "bef3960d18bd6e34509ccd26aae91a2e"
server_key_version: "encryptionkey@0"
I20260928 13:07:58.083798 9312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "0b92dcf9321f4885b76ceaf863e7ec46"
format_stamp: "Formatted at 2026-09-28 13:07:58 on dist-test-slave-p1lc"
server_key: "76040b959e75b10d5fcee33a72ffe678"
server_key_iv: "bef3960d18bd6e34509ccd26aae91a2e"
server_key_version: "encryptionkey@0"
I20260928 13:07:58.085558 9312 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 13:07:58.086436 9327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:58.086660 9312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:07:58.086781 9312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "0b92dcf9321f4885b76ceaf863e7ec46"
format_stamp: "Formatted at 2026-09-28 13:07:58 on dist-test-slave-p1lc"
server_key: "76040b959e75b10d5fcee33a72ffe678"
server_key_iv: "bef3960d18bd6e34509ccd26aae91a2e"
server_key_version: "encryptionkey@0"
I20260928 13:07:58.086850 9312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:07:58.104502 9312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:07:58.107046 9312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:07:58.107174 9312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:07:58.107396 9312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:07:58.107730 9312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:07:58.107769 9312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:58.107792 9312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:07:58.107805 9312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:58.114974 9312 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:42481
I20260928 13:07:58.115003 9440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:42481 every 8 connection(s)
I20260928 13:07:58.115448 9312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:07:58.117070 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 9312
I20260928 13:07:58.117175 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:07:58.117373 8546 external_mini_cluster.cc:1428] Setting key 5c2e21bfb45f9b2775e4c91058d5cc52
Sep 28 13:07:58 dist-test-slave-p1lc krb5kdc[9071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600878, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
I20260928 13:07:58.127689 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.116235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:42695 (local address 127.8.88.190:46767)
0928 13:07:58.116444 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:58.116447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:58.116523 (+ 76us) server_negotiation.cc:407] Connection header received
0928 13:07:58.117853 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:58.117854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:58.117866 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:58.117925 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:58.118691 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.119247 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.120494 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.120838 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.120960 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.123734 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:58.123737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:58.123738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:58.123747 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:58.125977 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:58.126509 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:58.126511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:58.126511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:58.126580 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:58.126879 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:58.126880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:58.126881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:58.127031 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:58.127090 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:58.127433 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 13:07:58.127598 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 13:07:58.127871 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.116211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46767 (local address 127.8.88.130:42695)
0928 13:07:58.116396 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 13:07:58.116439 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 13:07:58.117722 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:07:58.118002 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:07:58.118006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:07:58.118287 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:07:58.118605 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.118613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.119363 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.119364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.120374 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.120377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.120973 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.120974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.121129 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.121795 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:07:58.121808 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:07:58.123602 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:07:58.126087 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:58.126095 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:58.126098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:58.126421 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:58.126674 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:58.126676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:58.126676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:58.126806 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:58.127118 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:07:58.127125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:07:58.127277 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 13:07:58.127382 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 13:07:58.127525 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":107,"threads_started":1}
I20260928 13:07:58.128535 9441 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46767
I20260928 13:07:58.128666 9441 heartbeater.cc:464] Registering TS with master...
I20260928 13:07:58.128937 9441 heartbeater.cc:511] Master 127.8.88.190:46767 requested a full tablet report, sending...
I20260928 13:07:58.129385 9116 ts_manager.cc:194] Registered new tserver with Master: 0b92dcf9321f4885b76ceaf863e7ec46 (127.8.88.130:42481)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
I20260928 13:07:58.129963 9116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:42695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:07:58.148690 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46767
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:41543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:07:58.237300 9446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:07:58.237562 9446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:07:58.237607 9446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:07:58.238994 9446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:07:58.239075 9446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:07:58.239130 9446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:07:58.241026 9446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46767
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:07:58.241341 9446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:07:58.241612 9446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:07:58.247071 9452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:58.247123 9451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:58.247251 9454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:07:58.247478 9446 server_base.cc:1034] running on GCE node
I20260928 13:07:58.247730 9446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:07:58.248005 9446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:07:58.249226 9446 hybrid_clock.cc:648] HybridClock initialized: now 1790600878249203 us; error 43 us; skew 500 ppm
Sep 28 13:07:58 dist-test-slave-p1lc krb5kdc[9071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600878, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:07:58.252053 9446 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:07:58.252604 9446 webserver.cc:460] Webserver started at http://127.8.88.131:33567/ using document root <none> and password file <none>
I20260928 13:07:58.252859 9446 fs_manager.cc:359] Metadata directory not provided
I20260928 13:07:58.252987 9446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:07:58.253149 9446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:07:58.254268 9446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e"
format_stamp: "Formatted at 2026-09-28 13:07:58 on dist-test-slave-p1lc"
server_key: "7e644dc5e4962c10bc66fd3edcc468bb"
server_key_iv: "a83aaa1e1ef00c5d1e1da438afab479f"
server_key_version: "encryptionkey@0"
I20260928 13:07:58.254660 9446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e"
format_stamp: "Formatted at 2026-09-28 13:07:58 on dist-test-slave-p1lc"
server_key: "7e644dc5e4962c10bc66fd3edcc468bb"
server_key_iv: "a83aaa1e1ef00c5d1e1da438afab479f"
server_key_version: "encryptionkey@0"
I20260928 13:07:58.256157 9446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:07:58.257048 9461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:58.257278 9446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:07:58.257385 9446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e"
format_stamp: "Formatted at 2026-09-28 13:07:58 on dist-test-slave-p1lc"
server_key: "7e644dc5e4962c10bc66fd3edcc468bb"
server_key_iv: "a83aaa1e1ef00c5d1e1da438afab479f"
server_key_version: "encryptionkey@0"
I20260928 13:07:58.257474 9446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:07:58.279242 9446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:07:58.281847 9446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:07:58.281970 9446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:07:58.282153 9446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:07:58.282487 9446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:07:58.282526 9446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:58.282549 9446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:07:58.282601 9446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:58.289182 9446 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:40883
I20260928 13:07:58.289243 9574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:40883 every 8 connection(s)
I20260928 13:07:58.289619 9446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:07:58.294498 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 9446
I20260928 13:07:58.294617 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:07:58.294847 8546 external_mini_cluster.cc:1428] Setting key 544e67efcebc063a964cd714f6ee4291
Sep 28 13:07:58 dist-test-slave-p1lc krb5kdc[9071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600878, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:07:58.300233 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.290490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:59093 (local address 127.8.88.190:46767)
0928 13:07:58.290585 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:58.290588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:58.290785 (+ 197us) server_negotiation.cc:407] Connection header received
0928 13:07:58.291837 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:58.291838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:58.291852 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:58.291910 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:58.292589 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.293169 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.294205 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.294488 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.294567 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.297311 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:58.297313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:58.297313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:58.297320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:58.298899 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:58.299315 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:58.299317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:58.299317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:58.299374 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:58.299624 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:58.299625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:58.299626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:58.299761 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:58.299829 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:58.300077 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 13:07:58.300158 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 13:07:58.300498 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.290473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46767 (local address 127.8.88.131:59093)
0928 13:07:58.290642 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 13:07:58.290694 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 13:07:58.291697 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:07:58.291992 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:07:58.291997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:07:58.292283 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:07:58.292488 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.292493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.293275 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.293276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.294106 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.294108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.294702 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.294703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.294846 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.295581 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:07:58.295592 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:07:58.297217 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:07:58.298986 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:58.298994 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:58.298996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:58.299229 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:58.299437 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:58.299438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:58.299438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:58.299542 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:58.299829 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:07:58.299834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:07:58.299956 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 13:07:58.300040 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 13:07:58.300169 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260928 13:07:58.301070 9575 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46767
I20260928 13:07:58.301172 9575 heartbeater.cc:464] Registering TS with master...
I20260928 13:07:58.301332 9575 heartbeater.cc:511] Master 127.8.88.190:46767 requested a full tablet report, sending...
I20260928 13:07:58.301731 9116 ts_manager.cc:194] Registered new tserver with Master: a3bb788dc16f44b0bbceb2a8d2fd647e (127.8.88.131:40883)
I20260928 13:07:58.302145 9116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:59093
I20260928 13:07:58.306746 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:07:58.315150 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.308446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.8.88.190:46767)
0928 13:07:58.308524 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:58.308526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:58.308660 (+ 134us) server_negotiation.cc:407] Connection header received
0928 13:07:58.308700 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:58.308701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:58.308711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:58.308787 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:58.309409 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.309916 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.310617 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.310818 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.310869 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.312207 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:58.312209 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:58.312209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:58.312216 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:58.313854 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:58.314277 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:58.314278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:58.314278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:58.314334 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:58.314564 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:58.314565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:58.314565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:58.314717 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:58.314800 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:58.315007 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 13:07:58.315080 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 13:07:58.317734 9116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:07:58.319491 9116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:07:58.327072 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.322694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.8.88.129:39579)
0928 13:07:58.322907 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:58.322912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:58.322922 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:07:58.322968 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:58.322970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:58.322986 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:58.323073 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:07:58.323294 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.324463 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.325895 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.326588 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.326672 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.326738 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:58.326937 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 13:07:58.326986 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":103,"threads_started":1}
I20260928 13:07:58.327066 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.322513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:39579 (local address 127.0.0.1:48062)
0928 13:07:58.322699 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 13:07:58.322749 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 13:07:58.322825 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:07:58.323080 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:07:58.323083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:07:58.323085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:07:58.323172 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.323174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.324564 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.324565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.325826 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.325828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.326699 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.326700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.326768 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.326780 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 13:07:58.326910 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 13:07:58.326978 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":33,"spinlock_wait_cycles":1280,"thread_start_us":106,"threads_started":1}
I20260928 13:07:58.327538 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.322514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:40883 (local address 127.0.0.1:56690)
0928 13:07:58.322894 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 13:07:58.322906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:07:58.322960 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:07:58.323442 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:07:58.323443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:07:58.323445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:07:58.323483 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.323485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.324682 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.324683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.325726 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.325728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.327309 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.327310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.327374 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.327378 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:07:58.327429 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 13:07:58.327472 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":24,"thread_start_us":75,"threads_started":1}
I20260928 13:07:58.327817 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.322710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.8.88.131:40883)
0928 13:07:58.323136 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:58.323140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:58.323150 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:07:58.323185 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:58.323187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:58.323275 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:58.323350 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:07:58.323558 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.324574 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.325822 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.326588 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.326672 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.326749 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:58.327691 (+ 942us) server_negotiation.cc:299] Negotiation successful
0928 13:07:58.327743 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":92,"threads_started":1}
I20260928 13:07:58.327841 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.322777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:42481 (local address 127.0.0.1:53188)
0928 13:07:58.322981 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 13:07:58.322991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:07:58.323030 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:07:58.323434 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:07:58.323435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:07:58.323437 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:07:58.323493 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.323495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.324374 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.324375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.325812 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.325815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.327583 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.327583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.327650 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.327655 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:07:58.327702 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 13:07:58.327754 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":8,"thread_start_us":58,"threads_started":1}
I20260928 13:07:58.327955 9241 tablet_service.cc:1467] Processing CreateTablet for tablet 9d833849516f46bc8fd8a1ea1c0385cd (DEFAULT_TABLE table=test-table [id=e40a48f58cdc4291b9fe30a3fdb20a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:07:58.328171 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.322995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.8.88.130:42481)
0928 13:07:58.323215 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:58.323219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:58.323228 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:07:58.323263 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:58.323265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:58.323280 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:58.323354 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:07:58.323569 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.324264 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.326793 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.327421 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.327489 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.327546 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:58.328031 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 13:07:58.328097 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":99,"threads_started":1}
I20260928 13:07:58.328378 9241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d833849516f46bc8fd8a1ea1c0385cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:07:58.328531 9509 tablet_service.cc:1467] Processing CreateTablet for tablet 9d833849516f46bc8fd8a1ea1c0385cd (DEFAULT_TABLE table=test-table [id=e40a48f58cdc4291b9fe30a3fdb20a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:07:58.328814 9509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d833849516f46bc8fd8a1ea1c0385cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:07:58.328758 9375 tablet_service.cc:1467] Processing CreateTablet for tablet 9d833849516f46bc8fd8a1ea1c0385cd (DEFAULT_TABLE table=test-table [id=e40a48f58cdc4291b9fe30a3fdb20a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:07:58.329084 9375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d833849516f46bc8fd8a1ea1c0385cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:07:58.331676 9592 tablet_bootstrap.cc:492] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46: Bootstrap starting.
I20260928 13:07:58.332201 9593 tablet_bootstrap.cc:492] T 9d833849516f46bc8fd8a1ea1c0385cd P ddbe6fb6e05d40c4b56ff679b24cfde4: Bootstrap starting.
I20260928 13:07:58.332211 9594 tablet_bootstrap.cc:492] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e: Bootstrap starting.
I20260928 13:07:58.332422 9592 tablet_bootstrap.cc:654] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46: Neither blocks nor log segments found. Creating new log.
I20260928 13:07:58.332917 9594 tablet_bootstrap.cc:654] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e: Neither blocks nor log segments found. Creating new log.
I20260928 13:07:58.332918 9593 tablet_bootstrap.cc:654] T 9d833849516f46bc8fd8a1ea1c0385cd P ddbe6fb6e05d40c4b56ff679b24cfde4: Neither blocks nor log segments found. Creating new log.
I20260928 13:07:58.333037 9592 log.cc:824] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46: Log is configured to *not* fsync() on all Append() calls
I20260928 13:07:58.333765 9592 tablet_bootstrap.cc:492] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46: No bootstrap required, opened a new log
I20260928 13:07:58.333850 9592 ts_tablet_manager.cc:1397] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:07:58.333992 9594 log.cc:824] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e: Log is configured to *not* fsync() on all Append() calls
I20260928 13:07:58.333992 9593 log.cc:824] T 9d833849516f46bc8fd8a1ea1c0385cd P ddbe6fb6e05d40c4b56ff679b24cfde4: Log is configured to *not* fsync() on all Append() calls
I20260928 13:07:58.334729 9594 tablet_bootstrap.cc:492] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e: No bootstrap required, opened a new log
I20260928 13:07:58.334769 9593 tablet_bootstrap.cc:492] T 9d833849516f46bc8fd8a1ea1c0385cd P ddbe6fb6e05d40c4b56ff679b24cfde4: No bootstrap required, opened a new log
I20260928 13:07:58.334820 9594 ts_tablet_manager.cc:1397] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 13:07:58.334832 9593 ts_tablet_manager.cc:1397] T 9d833849516f46bc8fd8a1ea1c0385cd P ddbe6fb6e05d40c4b56ff679b24cfde4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 13:07:58.335989 9592 raft_consensus.cc:348] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39579 } } peers { permanent_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40883 } } peers { permanent_uuid: "0b92dcf9321f4885b76ceaf863e7ec46" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42481 } }
I20260928 13:07:58.336189 9592 raft_consensus.cc:374] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:07:58.336228 9592 raft_consensus.cc:729] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b92dcf9321f4885b76ceaf863e7ec46, State: Initialized, Role: FOLLOWER
I20260928 13:07:58.336392 9592 consensus_queue.cc:260] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39579 } } peers { permanent_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40883 } } peers { permanent_uuid: "0b92dcf9321f4885b76ceaf863e7ec46" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42481 } }
I20260928 13:07:58.336499 9594 raft_consensus.cc:348] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39579 } } peers { permanent_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40883 } } peers { permanent_uuid: "0b92dcf9321f4885b76ceaf863e7ec46" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42481 } }
I20260928 13:07:58.336702 9592 ts_tablet_manager.cc:1428] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:07:58.336717 9594 raft_consensus.cc:374] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:07:58.336759 9594 raft_consensus.cc:729] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3bb788dc16f44b0bbceb2a8d2fd647e, State: Initialized, Role: FOLLOWER
I20260928 13:07:58.336826 9441 heartbeater.cc:503] Master 127.8.88.190:46767 was elected leader, sending a full tablet report...
I20260928 13:07:58.336905 9594 consensus_queue.cc:260] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39579 } } peers { permanent_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40883 } } peers { permanent_uuid: "0b92dcf9321f4885b76ceaf863e7ec46" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42481 } }
I20260928 13:07:58.336947 9593 raft_consensus.cc:348] T 9d833849516f46bc8fd8a1ea1c0385cd P ddbe6fb6e05d40c4b56ff679b24cfde4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39579 } } peers { permanent_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40883 } } peers { permanent_uuid: "0b92dcf9321f4885b76ceaf863e7ec46" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42481 } }
I20260928 13:07:58.337111 9593 raft_consensus.cc:374] T 9d833849516f46bc8fd8a1ea1c0385cd P ddbe6fb6e05d40c4b56ff679b24cfde4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:07:58.337162 9593 raft_consensus.cc:729] T 9d833849516f46bc8fd8a1ea1c0385cd P ddbe6fb6e05d40c4b56ff679b24cfde4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbe6fb6e05d40c4b56ff679b24cfde4, State: Initialized, Role: FOLLOWER
I20260928 13:07:58.337209 9594 ts_tablet_manager.cc:1428] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:07:58.337335 9575 heartbeater.cc:503] Master 127.8.88.190:46767 was elected leader, sending a full tablet report...
I20260928 13:07:58.337316 9593 consensus_queue.cc:260] T 9d833849516f46bc8fd8a1ea1c0385cd P ddbe6fb6e05d40c4b56ff679b24cfde4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39579 } } peers { permanent_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40883 } } peers { permanent_uuid: "0b92dcf9321f4885b76ceaf863e7ec46" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42481 } }
I20260928 13:07:58.337626 9593 ts_tablet_manager.cc:1428] T 9d833849516f46bc8fd8a1ea1c0385cd P ddbe6fb6e05d40c4b56ff679b24cfde4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 13:07:58.337688 9307 heartbeater.cc:503] Master 127.8.88.190:46767 was elected leader, sending a full tablet report...
W20260928 13:07:58.366009 9442 tablet.cc:2369] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:07:58.371580 9599 raft_consensus.cc:482] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:07:58.371747 9599 raft_consensus.cc:504] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39579 } } peers { permanent_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40883 } } peers { permanent_uuid: "0b92dcf9321f4885b76ceaf863e7ec46" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42481 } }
I20260928 13:07:58.372136 9599 leader_election.cc:288] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddbe6fb6e05d40c4b56ff679b24cfde4 (127.8.88.129:39579), 0b92dcf9321f4885b76ceaf863e7ec46 (127.8.88.130:42481)
I20260928 13:07:58.375197 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.372420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:59235 (local address 127.8.88.129:39579)
0928 13:07:58.372555 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:58.372557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:58.372644 (+ 87us) server_negotiation.cc:407] Connection header received
0928 13:07:58.372673 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:58.372674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:58.372688 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:58.372776 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:07:58.372937 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.373400 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.374423 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.374839 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.374896 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.374904 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:58.375074 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 13:07:58.375122 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 13:07:58.375221 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.372340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:39579 (local address 127.8.88.131:59235)
0928 13:07:58.372552 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 13:07:58.372562 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:07:58.372621 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:07:58.372766 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:07:58.372766 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:07:58.372767 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:07:58.372858 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.372859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.373489 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.373490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.374331 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.374333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.374919 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.374921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.375006 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.375009 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:07:58.375050 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 13:07:58.375080 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":26,"thread_start_us":112,"threads_started":1}
I20260928 13:07:58.375260 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.372419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:39861 (local address 127.8.88.130:42481)
0928 13:07:58.372463 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:58.372466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:58.372478 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:07:58.372516 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:58.372516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:58.372529 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:58.372663 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:07:58.372843 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.373400 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.374489 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.374859 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.374952 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.374960 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:58.375142 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 13:07:58.375185 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:07:58.375499 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.372340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:42481 (local address 127.8.88.131:39861)
0928 13:07:58.372408 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 13:07:58.372420 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:07:58.372462 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:07:58.372671 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:07:58.372672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:07:58.372673 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:07:58.372755 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.372757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.373489 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.373490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.374388 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:58.374392 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.374966 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.374967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:58.375061 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.375065 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:07:58.375122 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 13:07:58.375159 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":52}
I20260928 13:07:58.375702 9261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d833849516f46bc8fd8a1ea1c0385cd" candidate_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4" is_pre_election: true
I20260928 13:07:58.375839 9261 raft_consensus.cc:2454] T 9d833849516f46bc8fd8a1ea1c0385cd P ddbe6fb6e05d40c4b56ff679b24cfde4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3bb788dc16f44b0bbceb2a8d2fd647e in term 0.
I20260928 13:07:58.375836 9395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d833849516f46bc8fd8a1ea1c0385cd" candidate_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b92dcf9321f4885b76ceaf863e7ec46" is_pre_election: true
I20260928 13:07:58.376009 9395 raft_consensus.cc:2454] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3bb788dc16f44b0bbceb2a8d2fd647e in term 0.
I20260928 13:07:58.376070 9463 leader_election.cc:302] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3bb788dc16f44b0bbceb2a8d2fd647e, ddbe6fb6e05d40c4b56ff679b24cfde4; no voters:
I20260928 13:07:58.376271 9599 raft_consensus.cc:2781] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:07:58.376319 9599 raft_consensus.cc:482] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:07:58.376335 9599 raft_consensus.cc:3037] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:07:58.377130 9599 raft_consensus.cc:504] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39579 } } peers { permanent_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40883 } } peers { permanent_uuid: "0b92dcf9321f4885b76ceaf863e7ec46" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42481 } }
I20260928 13:07:58.377274 9599 leader_election.cc:288] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [CANDIDATE]: Term 1 election: Requested vote from peers ddbe6fb6e05d40c4b56ff679b24cfde4 (127.8.88.129:39579), 0b92dcf9321f4885b76ceaf863e7ec46 (127.8.88.130:42481)
I20260928 13:07:58.377449 9395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d833849516f46bc8fd8a1ea1c0385cd" candidate_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b92dcf9321f4885b76ceaf863e7ec46"
I20260928 13:07:58.377434 9261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d833849516f46bc8fd8a1ea1c0385cd" candidate_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4"
I20260928 13:07:58.377517 9395 raft_consensus.cc:3037] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:07:58.377518 9261 raft_consensus.cc:3037] T 9d833849516f46bc8fd8a1ea1c0385cd P ddbe6fb6e05d40c4b56ff679b24cfde4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:07:58.378376 9261 raft_consensus.cc:2454] T 9d833849516f46bc8fd8a1ea1c0385cd P ddbe6fb6e05d40c4b56ff679b24cfde4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3bb788dc16f44b0bbceb2a8d2fd647e in term 1.
I20260928 13:07:58.378376 9395 raft_consensus.cc:2454] T 9d833849516f46bc8fd8a1ea1c0385cd P 0b92dcf9321f4885b76ceaf863e7ec46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3bb788dc16f44b0bbceb2a8d2fd647e in term 1.
I20260928 13:07:58.378585 9463 leader_election.cc:302] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3bb788dc16f44b0bbceb2a8d2fd647e, ddbe6fb6e05d40c4b56ff679b24cfde4; no voters:
I20260928 13:07:58.378711 9599 raft_consensus.cc:2781] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:07:58.378906 9599 raft_consensus.cc:686] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [term 1 LEADER]: Becoming Leader. State: Replica: a3bb788dc16f44b0bbceb2a8d2fd647e, State: Running, Role: LEADER
I20260928 13:07:58.379010 9599 consensus_queue.cc:237] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39579 } } peers { permanent_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40883 } } peers { permanent_uuid: "0b92dcf9321f4885b76ceaf863e7ec46" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42481 } }
I20260928 13:07:58.379926 9116 catalog_manager.cc:5495] T 9d833849516f46bc8fd8a1ea1c0385cd P a3bb788dc16f44b0bbceb2a8d2fd647e reported cstate change: term changed from 0 to 1, leader changed from <none> to a3bb788dc16f44b0bbceb2a8d2fd647e (127.8.88.131). New cstate: current_term: 1 leader_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbe6fb6e05d40c4b56ff679b24cfde4" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3bb788dc16f44b0bbceb2a8d2fd647e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b92dcf9321f4885b76ceaf863e7ec46" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42481 } health_report { overall_health: UNKNOWN } } }
Sep 28 13:07:58 dist-test-slave-p1lc krb5kdc[9071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.129@KRBTEST.COM
I20260928 13:07:58.392280 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.383820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.8.88.129:39579)
0928 13:07:58.383900 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:58.383902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:58.384022 (+ 120us) server_negotiation.cc:407] Connection header received
0928 13:07:58.384137 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:58.384138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:58.384156 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:58.384222 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:58.384800 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.385336 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.386074 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.386318 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.386413 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.389152 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:58.389166 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:58.389166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:58.389174 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:58.390932 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:58.391360 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:58.391364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:58.391365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:58.391427 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:58.391678 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:58.391679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:58.391679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:58.391847 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:58.391969 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:58.392074 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 13:07:58.392145 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 13:07:58 dist-test-slave-p1lc krb5kdc[9071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600878, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 13:07:58 dist-test-slave-p1lc krb5kdc[9071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600878, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.88.129@KRBTEST.COM
I20260928 13:07:58.417193 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.409171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.8.88.129:39579)
0928 13:07:58.409259 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:58.409262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:58.409393 (+ 131us) server_negotiation.cc:407] Connection header received
0928 13:07:58.409495 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:58.409496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:58.409507 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:58.409560 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:58.410154 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.410662 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.411435 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.411647 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.411746 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.414359 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:58.414361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:58.414361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:58.414369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:58.415775 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:58.416178 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:58.416179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:58.416179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:58.416282 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:58.416512 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:58.416513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:58.416513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:58.416627 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:58.416717 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:58.417026 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 13:07:58.417109 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 13:07:58 dist-test-slave-p1lc krb5kdc[9071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 13:07:58 dist-test-slave-p1lc krb5kdc[9071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.129@KRBTEST.COM
I20260928 13:07:58.441330 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:58.433647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48086 (local address 127.8.88.129:39579)
0928 13:07:58.433726 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:58.433728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:58.433986 (+ 258us) server_negotiation.cc:407] Connection header received
0928 13:07:58.434053 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:58.434054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:58.434068 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:58.434134 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:58.434789 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.435275 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.436030 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:58.436250 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:58.436309 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:58.438683 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:58.438684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:58.438685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:58.438692 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:58.440127 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:58.440506 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:58.440507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:58.440508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:58.440567 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:58.440763 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:58.440764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:58.440764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:58.440918 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:58.441007 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:58.441128 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 13:07:58.441192 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 13:07:58.441839 9221 tablet_service.cc:2908] Rejecting scan request for tablet 9d833849516f46bc8fd8a1ea1c0385cd: Uninitialized: clean time has not yet been initialized
I20260928 13:07:58.442816 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 9178
I20260928 13:07:58.448601 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 9312
I20260928 13:07:58.453569 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 9446
I20260928 13:07:58.458135 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 9084
2026-09-28T13:07:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2030 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:07:58 dist-test-slave-p1lc krb5kdc[9616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:07:58 dist-test-slave-p1lc krb5kdc[9616](info): set up 2 sockets
Sep 28 13:07:58 dist-test-slave-p1lc krb5kdc[9616](info): commencing operation
krb5kdc: starting...
W20260928 13:07:59.516963 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:07:59 dist-test-slave-p1lc krb5kdc[9616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:07:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:07:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:07:59.645275 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:43189
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:43189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:07:59.731127 9629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:07:59.731408 9629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:07:59.731448 9629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:07:59.732697 9629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:07:59.732775 9629 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 13:07:59.732798 9629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:07:59.732821 9629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:07:59.732852 9629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:07:59.732914 9629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:07:59.734527 9629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:43189
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:43189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:07:59.734838 9629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:07:59.735075 9629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:07:59.740176 9635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:59.740309 9637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:59.740141 9634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:07:59.740458 9629 server_base.cc:1034] running on GCE node
I20260928 13:07:59.740741 9629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:07:59.741081 9629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:07:59.742239 9629 hybrid_clock.cc:648] HybridClock initialized: now 1790600879742209 us; error 37 us; skew 500 ppm
Sep 28 13:07:59 dist-test-slave-p1lc krb5kdc[9616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600879, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:07:59.759549 9629 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:07:59.760183 9629 webserver.cc:460] Webserver started at http://127.8.88.190:42177/ using document root <none> and password file <none>
I20260928 13:07:59.760449 9629 fs_manager.cc:359] Metadata directory not provided
I20260928 13:07:59.760522 9629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:07:59.760706 9629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:07:59.761901 9629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "76889e4fd79a4cd39254b6c8ab98258a"
format_stamp: "Formatted at 2026-09-28 13:07:59 on dist-test-slave-p1lc"
server_key: "c7a4f19bfbecba3aa2d1f8d675a37ad4"
server_key_iv: "b8d46878dbaac3204f5e6fbda45b889f"
server_key_version: "encryptionkey@0"
I20260928 13:07:59.762276 9629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "76889e4fd79a4cd39254b6c8ab98258a"
format_stamp: "Formatted at 2026-09-28 13:07:59 on dist-test-slave-p1lc"
server_key: "c7a4f19bfbecba3aa2d1f8d675a37ad4"
server_key_iv: "b8d46878dbaac3204f5e6fbda45b889f"
server_key_version: "encryptionkey@0"
I20260928 13:07:59.763767 9629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:07:59.764735 9644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:59.765041 9629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:07:59.765173 9629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "76889e4fd79a4cd39254b6c8ab98258a"
format_stamp: "Formatted at 2026-09-28 13:07:59 on dist-test-slave-p1lc"
server_key: "c7a4f19bfbecba3aa2d1f8d675a37ad4"
server_key_iv: "b8d46878dbaac3204f5e6fbda45b889f"
server_key_version: "encryptionkey@0"
I20260928 13:07:59.765275 9629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:07:59.772816 9629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:07:59.775446 9629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:07:59.775625 9629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:07:59.780675 9629 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:43189
I20260928 13:07:59.780718 9706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:43189 every 8 connection(s)
I20260928 13:07:59.781140 9629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:07:59.781811 9707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:07:59.784088 9707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a: Bootstrap starting.
I20260928 13:07:59.784641 9707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a: Neither blocks nor log segments found. Creating new log.
I20260928 13:07:59.784961 9707 log.cc:824] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a: Log is configured to *not* fsync() on all Append() calls
I20260928 13:07:59.785681 9707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a: No bootstrap required, opened a new log
I20260928 13:07:59.787824 9707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76889e4fd79a4cd39254b6c8ab98258a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43189 } }
I20260928 13:07:59.787988 9707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:07:59.788012 9707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76889e4fd79a4cd39254b6c8ab98258a, State: Initialized, Role: FOLLOWER
I20260928 13:07:59.788184 9707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76889e4fd79a4cd39254b6c8ab98258a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43189 } }
I20260928 13:07:59.788275 9707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:07:59.788298 9707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:07:59.788335 9707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:07:59.788978 9707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76889e4fd79a4cd39254b6c8ab98258a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43189 } }
I20260928 13:07:59.789072 9707 leader_election.cc:302] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76889e4fd79a4cd39254b6c8ab98258a; no voters:
I20260928 13:07:59.789204 9707 leader_election.cc:288] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:07:59.789291 9710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:07:59.789505 9710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [term 1 LEADER]: Becoming Leader. State: Replica: 76889e4fd79a4cd39254b6c8ab98258a, State: Running, Role: LEADER
I20260928 13:07:59.789561 9707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:07:59.789606 9710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76889e4fd79a4cd39254b6c8ab98258a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43189 } }
I20260928 13:07:59.790051 9712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76889e4fd79a4cd39254b6c8ab98258a. Latest consensus state: current_term: 1 leader_uuid: "76889e4fd79a4cd39254b6c8ab98258a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76889e4fd79a4cd39254b6c8ab98258a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43189 } } }
I20260928 13:07:59.790164 9712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [sys.catalog]: This master's current role is: LEADER
I20260928 13:07:59.790041 9711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76889e4fd79a4cd39254b6c8ab98258a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76889e4fd79a4cd39254b6c8ab98258a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43189 } } }
I20260928 13:07:59.790355 9711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a [sys.catalog]: This master's current role is: LEADER
I20260928 13:07:59.790364 9718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:07:59.791066 9718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:07:59.791189 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 9629
I20260928 13:07:59.791285 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:07:59.791453 8546 external_mini_cluster.cc:1428] Setting key ed8edbb1d1c6901088fbd2fc5f8950fe
I20260928 13:07:59.793625 9718 catalog_manager.cc:1347] Generated new cluster ID: abb9bbd9cdc24aed84df4b08a626f72b
I20260928 13:07:59.793707 9718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:07:59 dist-test-slave-p1lc krb5kdc[9616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:07:59.803004 9718 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:07:59.803731 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:59.792074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.8.88.190:43189)
0928 13:07:59.792227 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:59.792236 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:59.792269 (+ 33us) server_negotiation.cc:407] Connection header received
0928 13:07:59.792405 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:59.792414 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:59.792478 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:59.792637 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:59.793301 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:59.794742 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:59.795805 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:59.796493 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:59.796619 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:59.799611 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:59.799617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:59.799617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:59.799631 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:59.801810 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:59.802348 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:59.802352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:59.802353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:59.802478 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:59.802724 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:59.802725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:59.802726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:59.803069 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:59.803280 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:59.803366 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 13:07:59.803516 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260928 13:07:59.803866 9718 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:07:59.821987 9718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a: Generated new TSK 0
I20260928 13:07:59.822263 9718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:07:59.856003 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:43189
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:07:59.942947 9733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:07:59.943221 9733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:07:59.943255 9733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:07:59.944768 9733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:07:59.944839 9733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:07:59.944923 9733 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 13:07:59.944967 9733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:07:59.946812 9733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.88.190:43189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:07:59.947099 9733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:07:59.947372 9733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:07:59.952991 9739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:59.952992 9741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:07:59.952991 9738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:07:59.953329 9733 server_base.cc:1034] running on GCE node
I20260928 13:07:59.953519 9733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:07:59.953739 9733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:07:59.954909 9733 hybrid_clock.cc:648] HybridClock initialized: now 1790600879954882 us; error 44 us; skew 500 ppm
Sep 28 13:07:59 dist-test-slave-p1lc krb5kdc[9616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600879, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:07:59.957780 9733 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:07:59.958359 9733 webserver.cc:460] Webserver started at http://127.8.88.129:37079/ using document root <none> and password file <none>
I20260928 13:07:59.958654 9733 fs_manager.cc:359] Metadata directory not provided
I20260928 13:07:59.958731 9733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:07:59.958886 9733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:07:59.960357 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "099fd9dc97514f099771e20f2bc4af1a"
format_stamp: "Formatted at 2026-09-28 13:07:59 on dist-test-slave-p1lc"
server_key: "9c87b6eaaade1b3a36f423c4ee698a3f"
server_key_iv: "8e30ccf7dacc77b33276c6b470f9ade6"
server_key_version: "encryptionkey@0"
I20260928 13:07:59.960774 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "099fd9dc97514f099771e20f2bc4af1a"
format_stamp: "Formatted at 2026-09-28 13:07:59 on dist-test-slave-p1lc"
server_key: "9c87b6eaaade1b3a36f423c4ee698a3f"
server_key_iv: "8e30ccf7dacc77b33276c6b470f9ade6"
server_key_version: "encryptionkey@0"
I20260928 13:07:59.962548 9733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:07:59.963841 9748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:59.964222 9733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:07:59.964448 9733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "099fd9dc97514f099771e20f2bc4af1a"
format_stamp: "Formatted at 2026-09-28 13:07:59 on dist-test-slave-p1lc"
server_key: "9c87b6eaaade1b3a36f423c4ee698a3f"
server_key_iv: "8e30ccf7dacc77b33276c6b470f9ade6"
server_key_version: "encryptionkey@0"
I20260928 13:07:59.964555 9733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:07:59.973613 9733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:07:59.975942 9733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:07:59.976122 9733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:07:59.976678 9733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:07:59.976747 9733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:07:59.976789 9733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:07:59.976833 9733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 13:07:59 dist-test-slave-p1lc krb5kdc[9616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600879, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:07:59.984206 9733 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:45173
I20260928 13:07:59.984360 9863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:45173 every 8 connection(s)
I20260928 13:07:59.984644 9733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:07:59.987494 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:59.977287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:55059 (local address 127.8.88.190:43189)
0928 13:07:59.977359 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:59.977361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:59.977578 (+ 217us) server_negotiation.cc:407] Connection header received
0928 13:07:59.978113 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:59.978115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:59.978129 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:59.978205 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:59.978989 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:59.979847 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:59.981043 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:59.981420 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:59.981505 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:59.984708 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:59.984710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:59.984711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:59.984720 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:59.986181 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:59.986603 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:59.986604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:59.986604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:59.986661 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:59.986888 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:59.986889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:59.986889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:59.986998 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:59.987047 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:59.987336 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 13:07:59.987426 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 13:07:59.987548 9757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:59.977282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43189 (local address 127.8.88.129:55059)
0928 13:07:59.977457 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 13:07:59.977495 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 13:07:59.977994 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:07:59.978344 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:07:59.978348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:07:59.978676 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:07:59.978874 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:59.978879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:59.979933 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:59.979934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:59.980934 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:59.980937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:59.981531 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:59.981532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:59.981698 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:59.982469 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:07:59.982477 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:07:59.984564 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:07:59.986270 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:59.986277 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:59.986279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:59.986528 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:59.986717 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:59.986717 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:59.986718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:59.986832 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:59.987038 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:07:59.987040 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:07:59.987161 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 13:07:59.987244 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 13:07:59.987381 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260928 13:07:59.988399 9864 heartbeater.cc:347] Connected to a master server at 127.8.88.190:43189
I20260928 13:07:59.988504 9864 heartbeater.cc:464] Registering TS with master...
I20260928 13:07:59.988722 9864 heartbeater.cc:511] Master 127.8.88.190:43189 requested a full tablet report, sending...
I20260928 13:07:59.989439 9660 ts_manager.cc:194] Registered new tserver with Master: 099fd9dc97514f099771e20f2bc4af1a (127.8.88.129:45173)
I20260928 13:07:59.990316 9660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:55059
I20260928 13:07:59.991633 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 9733
I20260928 13:07:59.991755 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:07:59.991923 8546 external_mini_cluster.cc:1428] Setting key b6ad9cc080f431101cde09eec443a015
I20260928 13:07:59.997776 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:59.989715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:52351 (local address 127.8.88.190:43189)
0928 13:07:59.989768 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 13:07:59.989770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:07:59.989913 (+ 143us) server_negotiation.cc:407] Connection header received
0928 13:07:59.989998 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:07:59.989999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:07:59.990010 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:07:59.990064 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:07:59.990462 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:59.991328 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:59.992211 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:07:59.992487 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:07:59.992559 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:59.993923 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:07:59.993924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:07:59.993925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:07:59.993933 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:07:59.996145 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:59.996600 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:59.996601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:59.996602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:59.996702 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:07:59.996954 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:07:59.996955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:07:59.996956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:07:59.997150 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:07:59.997212 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 13:07:59.997544 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 13:07:59.997663 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 13:07:59.997786 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:07:59.989648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43189 (local address 127.8.88.129:52351)
0928 13:07:59.989838 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 13:07:59.989848 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:07:59.989932 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:07:59.990067 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:07:59.990067 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:07:59.990310 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:07:59.990367 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:59.990369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:59.991439 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:59.991441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:59.992115 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:07:59.992118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:07:59.992585 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:07:59.992586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:07:59.992669 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:07:59.993327 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:07:59.993332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:07:59.993811 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:07:59.996249 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:59.996250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:59.996251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:59.996510 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:59.996784 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:07:59.996785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:07:59.996785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:07:59.996875 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:07:59.997216 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:07:59.997217 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:07:59.997270 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 13:07:59.997507 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 13:07:59.997660 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:08:00.026443 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:43189
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:00.111840 9873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:00.112071 9873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:00.112099 9873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:00.113507 9873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:00.113565 9873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:00.113590 9873 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 13:08:00.113610 9873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:08:00.115195 9873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.88.190:43189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:00.115485 9873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:00.115744 9873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:00.121169 9881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:00.121171 9878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:00.121177 9879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:00.121538 9873 server_base.cc:1034] running on GCE node
I20260928 13:08:00.121803 9873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:00.122133 9873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:00.123342 9873 hybrid_clock.cc:648] HybridClock initialized: now 1790600880123327 us; error 35 us; skew 500 ppm
Sep 28 13:08:00 dist-test-slave-p1lc krb5kdc[9616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600880, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:00.126369 9873 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:08:00.126920 9873 webserver.cc:460] Webserver started at http://127.8.88.130:38701/ using document root <none> and password file <none>
I20260928 13:08:00.127171 9873 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:00.127245 9873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:00.127398 9873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:00.128530 9873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "c424a71075a745f883767f26c89805a6"
format_stamp: "Formatted at 2026-09-28 13:08:00 on dist-test-slave-p1lc"
server_key: "6a64bcf87aa03fb956a2c9a5b34a65ba"
server_key_iv: "e177ab87c64fd791fc98587d96ec82f0"
server_key_version: "encryptionkey@0"
I20260928 13:08:00.128985 9873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "c424a71075a745f883767f26c89805a6"
format_stamp: "Formatted at 2026-09-28 13:08:00 on dist-test-slave-p1lc"
server_key: "6a64bcf87aa03fb956a2c9a5b34a65ba"
server_key_iv: "e177ab87c64fd791fc98587d96ec82f0"
server_key_version: "encryptionkey@0"
I20260928 13:08:00.130421 9873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:00.131412 9888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:00.131682 9873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:00.131793 9873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "c424a71075a745f883767f26c89805a6"
format_stamp: "Formatted at 2026-09-28 13:08:00 on dist-test-slave-p1lc"
server_key: "6a64bcf87aa03fb956a2c9a5b34a65ba"
server_key_iv: "e177ab87c64fd791fc98587d96ec82f0"
server_key_version: "encryptionkey@0"
I20260928 13:08:00.131912 9873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:00.147457 9873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:00.150065 9873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:00.150249 9873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:00.150898 9873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:00.150955 9873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:00.150990 9873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:00.151034 9873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 13:08:00 dist-test-slave-p1lc krb5kdc[9616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600880, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:00.158627 9873 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:40121
I20260928 13:08:00.159013 9873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:08:00.159024 10003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:40121 every 8 connection(s)
I20260928 13:08:00.161041 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.151336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:45249 (local address 127.8.88.190:43189)
0928 13:08:00.151409 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.151412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.151665 (+ 253us) server_negotiation.cc:407] Connection header received
0928 13:08:00.152200 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.152201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.152215 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.152294 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:00.153000 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.153845 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.155042 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.155249 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.155338 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.157792 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:00.157793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:00.157794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:00.157802 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:00.159584 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:00.159982 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:00.159983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:00.159983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:00.160041 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:00.160306 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:00.160307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:00.160308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:00.160438 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:00.160515 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.160834 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.160948 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 13:08:00.161262 9897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.151339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43189 (local address 127.8.88.130:45249)
0928 13:08:00.151522 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.151565 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.152064 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.152386 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.152390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.152625 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:00.152842 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.152846 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.153957 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.153958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.154935 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.154938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.155351 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.155352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.155488 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.156107 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:00.156124 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:00.157671 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:00.159656 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:00.159662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:00.159665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:00.159898 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:00.160117 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:00.160118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:00.160118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:00.160234 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:00.160505 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:00.160512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:00.160657 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.160807 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.161004 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":113,"threads_started":1}
I20260928 13:08:00.161748 10004 heartbeater.cc:347] Connected to a master server at 127.8.88.190:43189
I20260928 13:08:00.161831 10004 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:00.162016 10004 heartbeater.cc:511] Master 127.8.88.190:43189 requested a full tablet report, sending...
I20260928 13:08:00.162405 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 9873
I20260928 13:08:00.162490 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:08:00.162716 8546 external_mini_cluster.cc:1428] Setting key 404e96d2508a15937c88e38f99604f90
I20260928 13:08:00.162943 9661 ts_manager.cc:194] Registered new tserver with Master: c424a71075a745f883767f26c89805a6 (127.8.88.130:40121)
I20260928 13:08:00.163326 9661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:45249
I20260928 13:08:00.170717 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.162868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43189 (local address 127.8.88.130:36173)
0928 13:08:00.163333 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.163346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.163422 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.163538 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.163539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.163756 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:00.163823 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.163826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.164853 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.164854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.165856 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.165860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.166362 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.166362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.166424 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.166930 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:00.166934 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:00.167309 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:00.169484 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:00.169485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:00.169485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:00.169711 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:00.169975 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:00.169976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:00.169977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:00.170047 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:00.170440 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:00.170441 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:00.170486 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.170552 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.170642 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":114,"threads_started":1}
I20260928 13:08:00.170799 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.162863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:36173 (local address 127.8.88.190:43189)
0928 13:08:00.162905 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.162908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.163397 (+ 489us) server_negotiation.cc:407] Connection header received
0928 13:08:00.163476 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.163477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.163487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.163535 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:00.163918 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.164663 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.165948 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.166273 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.166348 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.167405 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:00.167406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:00.167407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:00.167415 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:00.169387 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:00.169798 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:00.169800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:00.169800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:00.169871 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:00.170128 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:00.170129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:00.170130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:00.170349 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:00.170429 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.170605 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.170684 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:08:00.193145 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:43189
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:35093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:00.278487 10013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:00.278719 10013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:00.278748 10013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:00.279923 10013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:00.279963 10013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:00.279983 10013 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 13:08:00.280001 10013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:08:00.281513 10013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:35093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.88.190:43189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:00.281773 10013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:00.281975 10013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:00.287223 10019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:00.287249 10021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:00.287326 10018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:00.287405 10013 server_base.cc:1034] running on GCE node
I20260928 13:08:00.287703 10013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:00.288018 10013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:00.289286 10013 hybrid_clock.cc:648] HybridClock initialized: now 1790600880289284 us; error 45 us; skew 500 ppm
Sep 28 13:08:00 dist-test-slave-p1lc krb5kdc[9616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600880, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:00.292454 10013 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:08:00.293051 10013 webserver.cc:460] Webserver started at http://127.8.88.131:39099/ using document root <none> and password file <none>
I20260928 13:08:00.293316 10013 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:00.293394 10013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:00.293551 10013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:00.294678 10013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "26da8b8d723a4fd9af6945816834314a"
format_stamp: "Formatted at 2026-09-28 13:08:00 on dist-test-slave-p1lc"
server_key: "171a5e8060b8b1518222978fb932c7e2"
server_key_iv: "587b6a63753e99208d810f13e9f7cee2"
server_key_version: "encryptionkey@0"
I20260928 13:08:00.295076 10013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "26da8b8d723a4fd9af6945816834314a"
format_stamp: "Formatted at 2026-09-28 13:08:00 on dist-test-slave-p1lc"
server_key: "171a5e8060b8b1518222978fb932c7e2"
server_key_iv: "587b6a63753e99208d810f13e9f7cee2"
server_key_version: "encryptionkey@0"
I20260928 13:08:00.296530 10013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:08:00.297464 10028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:00.297703 10013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:08:00.297842 10013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "26da8b8d723a4fd9af6945816834314a"
format_stamp: "Formatted at 2026-09-28 13:08:00 on dist-test-slave-p1lc"
server_key: "171a5e8060b8b1518222978fb932c7e2"
server_key_iv: "587b6a63753e99208d810f13e9f7cee2"
server_key_version: "encryptionkey@0"
I20260928 13:08:00.297933 10013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:00.315874 10013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:00.318877 10013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:00.319062 10013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:00.319608 10013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:00.319675 10013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:00.319713 10013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:00.319748 10013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 13:08:00 dist-test-slave-p1lc krb5kdc[9616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600880, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:00.327592 10013 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:38309
I20260928 13:08:00.328037 10013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:08:00.328040 10143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:38309 every 8 connection(s)
I20260928 13:08:00.328670 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 10013
I20260928 13:08:00.328766 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:08:00.328969 8546 external_mini_cluster.cc:1428] Setting key 3d3074aa4a929b7ba808bda59318edc8
I20260928 13:08:00.330638 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.320339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:52553 (local address 127.8.88.190:43189)
0928 13:08:00.320405 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.320408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.320586 (+ 178us) server_negotiation.cc:407] Connection header received
0928 13:08:00.321222 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.321224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.321237 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.321283 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:00.322156 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.323028 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.324215 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.324437 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.324525 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.326897 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:00.326899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:00.326899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:00.326908 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:00.328777 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:00.329316 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:00.329318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:00.329318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:00.329414 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:00.329753 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:00.329754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:00.329755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:00.329889 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:00.329944 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.330480 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.330570 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 13:08:00.330747 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.320266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43189 (local address 127.8.88.131:52553)
0928 13:08:00.320465 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.320503 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.321120 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.321361 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.321365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.321747 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:00.322002 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.322007 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.323167 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.323168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.324101 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.324103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.324548 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.324550 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.324677 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.325333 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:00.325342 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:00.326785 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:00.328903 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:00.328910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:00.328912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:00.329238 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:00.329488 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:00.329489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:00.329489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:00.329664 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:00.330036 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:00.330043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:00.330249 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.330368 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.330543 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260928 13:08:00.331290 10144 heartbeater.cc:347] Connected to a master server at 127.8.88.190:43189
I20260928 13:08:00.331387 10144 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:00.331583 10144 heartbeater.cc:511] Master 127.8.88.190:43189 requested a full tablet report, sending...
I20260928 13:08:00.331945 9661 ts_manager.cc:194] Registered new tserver with Master: 26da8b8d723a4fd9af6945816834314a (127.8.88.131:38309)
I20260928 13:08:00.332347 9661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:52553
I20260928 13:08:00.338639 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.332470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43189 (local address 127.8.88.131:52345)
0928 13:08:00.332600 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.332612 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.332670 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.332807 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.332808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.333035 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:00.333096 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.333098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.333799 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.333800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.334363 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.334365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.334706 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.334707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.334834 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.335293 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:00.335298 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:00.335587 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:00.337548 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:00.337549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:00.337550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:00.337732 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:00.337947 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:00.337948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:00.337948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:00.338019 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:00.338280 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:00.338281 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:00.338336 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.338412 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.338522 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260928 13:08:00.338676 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.332508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:52345 (local address 127.8.88.190:43189)
0928 13:08:00.332553 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.332555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.332656 (+ 101us) server_negotiation.cc:407] Connection header received
0928 13:08:00.332733 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.332734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.332744 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.332802 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:00.333173 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.333693 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.334445 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.334636 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.334692 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.335674 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:00.335675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:00.335675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:00.335681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:00.337432 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:00.337812 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:00.337814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:00.337814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:00.337878 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:00.338087 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:00.338088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:00.338088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:00.338210 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:00.338289 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.338437 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.338519 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:08:00.341441 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 13:08:00 dist-test-slave-p1lc krb5kdc[9616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600880, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 13:08:00 dist-test-slave-p1lc krb5kdc[9616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600880, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:00.366688 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.358367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.8.88.190:43189)
0928 13:08:00.358440 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.358442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.358450 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:08:00.358483 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.358483 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.358492 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.358559 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:00.359225 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.359725 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.360501 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.360702 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.360789 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.363760 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:00.363762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:00.363762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:00.363771 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:00.365269 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:00.365695 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:00.365696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:00.365696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:00.365761 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:00.366103 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:00.366105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:00.366105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:00.366237 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:00.366372 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.366496 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.366582 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 13:08:00.368835 9661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:08:00.370529 9661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:00.377893 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.373447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:38309 (local address 127.0.0.1:60134)
0928 13:08:00.373993 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.374004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.374062 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.374437 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.374438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.374439 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:00.374500 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.374502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.375348 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.375349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.376664 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.376667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.377547 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.377549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.377637 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.377653 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.377722 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.377780 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":100,"threads_started":1}
I20260928 13:08:00.378028 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.373428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:45173 (local address 127.0.0.1:54626)
0928 13:08:00.373615 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.373660 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.373730 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.374282 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.374285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.374288 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:00.374366 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.374369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.375708 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.375709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.376765 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.376767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.377792 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.377793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.377858 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.377862 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.377909 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.377957 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":57,"threads_started":1}
I20260928 13:08:00.378178 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.373659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:40121 (local address 127.0.0.1:57568)
0928 13:08:00.374374 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.374384 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.374436 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.374661 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.374662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.374664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:00.374713 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.374714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.375931 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.375932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.376949 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.376952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.377914 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.377915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.377980 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.377985 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.378039 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.378093 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":74,"threads_started":1}
I20260928 13:08:00.378305 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.373769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60134 (local address 127.8.88.131:38309)
0928 13:08:00.374097 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.374102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.374115 (+ 13us) server_negotiation.cc:407] Connection header received
0928 13:08:00.374155 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.374157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.374173 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.374257 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:00.374560 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.375254 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.376770 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.377440 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.377544 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.377624 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.378128 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.378175 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":124,"threads_started":1}
I20260928 13:08:00.378567 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.373542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.8.88.129:45173)
0928 13:08:00.374023 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.374028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.374039 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:08:00.374083 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.374086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.374104 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.374276 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:00.374463 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.375610 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.376859 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.377649 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.377793 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.377865 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.377935 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.378471 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":109,"threads_started":1}
I20260928 13:08:00.378738 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.373794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.8.88.130:40121)
0928 13:08:00.374387 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.374391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.374428 (+ 37us) server_negotiation.cc:407] Connection header received
0928 13:08:00.374500 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.374503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.374519 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.374593 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:00.374796 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.375850 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.377037 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.377768 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.377829 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.377876 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.378349 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.378631 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":100,"threads_started":1}
I20260928 13:08:00.379079 10078 tablet_service.cc:1467] Processing CreateTablet for tablet f9d1b0691803445897da65bad9ef40b8 (DEFAULT_TABLE table=test-table [id=86570d59869d4000b09d725c8cef7281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:00.379305 9798 tablet_service.cc:1467] Processing CreateTablet for tablet f9d1b0691803445897da65bad9ef40b8 (DEFAULT_TABLE table=test-table [id=86570d59869d4000b09d725c8cef7281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:00.379470 10078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d1b0691803445897da65bad9ef40b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:00.379369 9938 tablet_service.cc:1467] Processing CreateTablet for tablet f9d1b0691803445897da65bad9ef40b8 (DEFAULT_TABLE table=test-table [id=86570d59869d4000b09d725c8cef7281]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:00.379670 9938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d1b0691803445897da65bad9ef40b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:00.379680 9798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9d1b0691803445897da65bad9ef40b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:00.382620 10167 tablet_bootstrap.cc:492] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a: Bootstrap starting.
I20260928 13:08:00.383126 10168 tablet_bootstrap.cc:492] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a: Bootstrap starting.
I20260928 13:08:00.383549 10167 tablet_bootstrap.cc:654] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:00.383939 10167 log.cc:824] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:00.384256 10168 tablet_bootstrap.cc:654] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:00.384315 10169 tablet_bootstrap.cc:492] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6: Bootstrap starting.
I20260928 13:08:00.384581 10168 log.cc:824] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:00.384775 10167 tablet_bootstrap.cc:492] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a: No bootstrap required, opened a new log
I20260928 13:08:00.384865 10167 ts_tablet_manager.cc:1397] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:00.384989 10169 tablet_bootstrap.cc:654] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:00.385277 10169 log.cc:824] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:00.386054 10169 tablet_bootstrap.cc:492] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6: No bootstrap required, opened a new log
I20260928 13:08:00.386121 10169 ts_tablet_manager.cc:1397] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 13:08:00.386608 10167 raft_consensus.cc:348] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } }
I20260928 13:08:00.386755 10167 raft_consensus.cc:374] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:00.386780 10167 raft_consensus.cc:729] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099fd9dc97514f099771e20f2bc4af1a, State: Initialized, Role: FOLLOWER
I20260928 13:08:00.386898 10167 consensus_queue.cc:260] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } }
I20260928 13:08:00.387184 10167 ts_tablet_manager.cc:1428] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:08:00.387231 9864 heartbeater.cc:503] Master 127.8.88.190:43189 was elected leader, sending a full tablet report...
I20260928 13:08:00.388010 10169 raft_consensus.cc:348] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } }
I20260928 13:08:00.388212 10169 raft_consensus.cc:374] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:00.388258 10169 raft_consensus.cc:729] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c424a71075a745f883767f26c89805a6, State: Initialized, Role: FOLLOWER
I20260928 13:08:00.388407 10169 consensus_queue.cc:260] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } }
I20260928 13:08:00.388737 10169 ts_tablet_manager.cc:1428] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 13:08:00.388818 10004 heartbeater.cc:503] Master 127.8.88.190:43189 was elected leader, sending a full tablet report...
I20260928 13:08:00.389309 10168 tablet_bootstrap.cc:492] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a: No bootstrap required, opened a new log
I20260928 13:08:00.389401 10168 ts_tablet_manager.cc:1397] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 13:08:00.391093 10168 raft_consensus.cc:348] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } }
I20260928 13:08:00.391273 10168 raft_consensus.cc:374] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:00.391315 10168 raft_consensus.cc:729] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26da8b8d723a4fd9af6945816834314a, State: Initialized, Role: FOLLOWER
I20260928 13:08:00.391439 10168 consensus_queue.cc:260] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } }
I20260928 13:08:00.391727 10168 ts_tablet_manager.cc:1428] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:08:00.391799 10144 heartbeater.cc:503] Master 127.8.88.190:43189 was elected leader, sending a full tablet report...
W20260928 13:08:00.409703 10005 tablet.cc:2369] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:08:00.485918 9865 tablet.cc:2369] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:08:00.578764 10145 tablet.cc:2369] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:08:00.626876 10173 raft_consensus.cc:482] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:00.627017 10173 raft_consensus.cc:504] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } }
I20260928 13:08:00.627336 10173 leader_election.cc:288] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26da8b8d723a4fd9af6945816834314a (127.8.88.131:38309), c424a71075a745f883767f26c89805a6 (127.8.88.130:40121)
I20260928 13:08:00.630635 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.628017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:59089 (local address 127.8.88.130:40121)
0928 13:08:00.628059 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.628062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.628068 (+ 6us) server_negotiation.cc:407] Connection header received
0928 13:08:00.628113 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.628114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.628123 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.628167 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:00.628427 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.628905 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.629895 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.630268 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.630325 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.630333 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.630509 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.630559 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:08:00.630705 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.627775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:44535 (local address 127.8.88.131:38309)
0928 13:08:00.627854 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.627856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.627865 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:08:00.627890 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.627891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.627921 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.628000 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:00.628387 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.628946 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.629951 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.630325 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.630412 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.630421 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.630591 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.630637 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 13:08:00.630724 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.627608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:38309 (local address 127.8.88.129:44535)
0928 13:08:00.627745 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.627758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.627823 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.628195 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.628196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.628199 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:00.628298 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.628301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.629065 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.629066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.629825 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.629828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.630425 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.630425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.630501 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.630505 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.630564 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.630604 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260928 13:08:00.630862 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.627854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:40121 (local address 127.8.88.129:59089)
0928 13:08:00.627967 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.627978 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.628030 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.628239 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.628240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.628242 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:00.628306 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.628308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.628976 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.628977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.629802 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.629804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.630349 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.630350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.630430 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.630435 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.630492 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.630528 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":47,"threads_started":1}
I20260928 13:08:00.631381 10098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d1b0691803445897da65bad9ef40b8" candidate_uuid: "099fd9dc97514f099771e20f2bc4af1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26da8b8d723a4fd9af6945816834314a" is_pre_election: true
I20260928 13:08:00.631453 9958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d1b0691803445897da65bad9ef40b8" candidate_uuid: "099fd9dc97514f099771e20f2bc4af1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c424a71075a745f883767f26c89805a6" is_pre_election: true
I20260928 13:08:00.631546 10098 raft_consensus.cc:2454] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 099fd9dc97514f099771e20f2bc4af1a in term 0.
I20260928 13:08:00.631649 9958 raft_consensus.cc:2454] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 099fd9dc97514f099771e20f2bc4af1a in term 0.
I20260928 13:08:00.631814 9751 leader_election.cc:302] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 099fd9dc97514f099771e20f2bc4af1a, 26da8b8d723a4fd9af6945816834314a; no voters:
I20260928 13:08:00.631966 10173 raft_consensus.cc:2781] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:00.632025 10173 raft_consensus.cc:482] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:00.632048 10173 raft_consensus.cc:3037] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:00.632826 10173 raft_consensus.cc:504] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } }
I20260928 13:08:00.633018 10173 leader_election.cc:288] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [CANDIDATE]: Term 1 election: Requested vote from peers 26da8b8d723a4fd9af6945816834314a (127.8.88.131:38309), c424a71075a745f883767f26c89805a6 (127.8.88.130:40121)
I20260928 13:08:00.633275 10098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d1b0691803445897da65bad9ef40b8" candidate_uuid: "099fd9dc97514f099771e20f2bc4af1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26da8b8d723a4fd9af6945816834314a"
I20260928 13:08:00.633282 9958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9d1b0691803445897da65bad9ef40b8" candidate_uuid: "099fd9dc97514f099771e20f2bc4af1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c424a71075a745f883767f26c89805a6"
I20260928 13:08:00.633386 9958 raft_consensus.cc:3037] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:00.633392 10098 raft_consensus.cc:3037] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:00.634193 9958 raft_consensus.cc:2454] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 099fd9dc97514f099771e20f2bc4af1a in term 1.
I20260928 13:08:00.634366 10098 raft_consensus.cc:2454] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 099fd9dc97514f099771e20f2bc4af1a in term 1.
I20260928 13:08:00.634429 9751 leader_election.cc:302] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 099fd9dc97514f099771e20f2bc4af1a, c424a71075a745f883767f26c89805a6; no voters:
I20260928 13:08:00.634531 10173 raft_consensus.cc:2781] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:00.634703 10173 raft_consensus.cc:686] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [term 1 LEADER]: Becoming Leader. State: Replica: 099fd9dc97514f099771e20f2bc4af1a, State: Running, Role: LEADER
I20260928 13:08:00.634826 10173 consensus_queue.cc:237] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } }
I20260928 13:08:00.635703 9661 catalog_manager.cc:5495] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 099fd9dc97514f099771e20f2bc4af1a (127.8.88.129). New cstate: current_term: 1 leader_uuid: "099fd9dc97514f099771e20f2bc4af1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } health_report { overall_health: UNKNOWN } } }
I20260928 13:08:00.667537 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.664836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54632 (local address 127.8.88.129:45173)
0928 13:08:00.664917 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.664919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.664930 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:08:00.665027 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.665028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.665054 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.665130 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:00.665292 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.665793 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.666640 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.666844 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.666933 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.667163 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:00.667277 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:00.667380 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.667423 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.667447 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 13:08:00.670024 10098 raft_consensus.cc:1261] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a [term 1 FOLLOWER]: Refusing update from remote peer 099fd9dc97514f099771e20f2bc4af1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:00.670024 9958 raft_consensus.cc:1261] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6 [term 1 FOLLOWER]: Refusing update from remote peer 099fd9dc97514f099771e20f2bc4af1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:00.670454 10178 consensus_queue.cc:1035] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:00.670614 10173 consensus_queue.cc:1035] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:00.673550 10181 mvcc.cc:205] Tried to move back new op lower bound from 7334301207221157888 to 7334301207080882176. Current Snapshot: MvccSnapshot[applied={T|T < 7334301207221157888 or (T in {7334301207221157888})}]
I20260928 13:08:00.673667 10182 mvcc.cc:205] Tried to move back new op lower bound from 7334301207221157888 to 7334301207080882176. Current Snapshot: MvccSnapshot[applied={T|T < 7334301207221157888}]
I20260928 13:08:00.673959 10183 mvcc.cc:205] Tried to move back new op lower bound from 7334301207221157888 to 7334301207080882176. Current Snapshot: MvccSnapshot[applied={T|T < 7334301207221157888}]
I20260928 13:08:00.676901 9660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:00.677033 9660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:00.677874 9660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a: Sending DeleteTablet for 3 replicas of tablet f9d1b0691803445897da65bad9ef40b8
I20260928 13:08:00.678321 9938 tablet_service.cc:1514] Processing DeleteTablet for tablet f9d1b0691803445897da65bad9ef40b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:00 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:57568
I20260928 13:08:00.678341 10078 tablet_service.cc:1514] Processing DeleteTablet for tablet f9d1b0691803445897da65bad9ef40b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:00 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:60134
I20260928 13:08:00.678347 9798 tablet_service.cc:1514] Processing DeleteTablet for tablet f9d1b0691803445897da65bad9ef40b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:00 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:54626
I20260928 13:08:00.678817 10192 tablet_replica.cc:330] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a: stopping tablet replica
I20260928 13:08:00.678932 9660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:08:00.679105 9660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:00.679378 10192 raft_consensus.cc:2229] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:08:00.679553 10192 raft_consensus.cc:2258] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:00.680518 10192 ts_tablet_manager.cc:1905] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:00.680677 9798 tablet_service.cc:1467] Processing CreateTablet for tablet 30f2b96f57a9492dbb37258c07c8703a (DEFAULT_TABLE table=test-table [id=eed41887ab32440d8e37fa01c38c570b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:00.680804 9798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f2b96f57a9492dbb37258c07c8703a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:00.680804 9938 tablet_service.cc:1467] Processing CreateTablet for tablet 30f2b96f57a9492dbb37258c07c8703a (DEFAULT_TABLE table=test-table [id=eed41887ab32440d8e37fa01c38c570b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:00.680958 9938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f2b96f57a9492dbb37258c07c8703a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:00.681162 10078 tablet_service.cc:1467] Processing CreateTablet for tablet 30f2b96f57a9492dbb37258c07c8703a (DEFAULT_TABLE table=test-table [id=eed41887ab32440d8e37fa01c38c570b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:00.681295 10078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30f2b96f57a9492dbb37258c07c8703a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:00.682097 10169 tablet_bootstrap.cc:492] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6: Bootstrap starting.
I20260928 13:08:00.682331 10167 tablet_bootstrap.cc:492] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a: Bootstrap starting.
I20260928 13:08:00.682520 10169 tablet_bootstrap.cc:654] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:00.682767 10167 tablet_bootstrap.cc:654] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:00.682890 10168 tablet_bootstrap.cc:492] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a: Bootstrap starting.
I20260928 13:08:00.683307 10168 tablet_bootstrap.cc:654] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:00.683413 10167 tablet_bootstrap.cc:492] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a: No bootstrap required, opened a new log
I20260928 13:08:00.683447 10167 ts_tablet_manager.cc:1397] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:00.683560 10167 raft_consensus.cc:348] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.683660 10167 raft_consensus.cc:374] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:00.683694 10167 raft_consensus.cc:729] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099fd9dc97514f099771e20f2bc4af1a, State: Initialized, Role: FOLLOWER
I20260928 13:08:00.683743 10167 consensus_queue.cc:260] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.683846 10167 ts_tablet_manager.cc:1428] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:00.685256 10190 tablet_replica.cc:330] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6: stopping tablet replica
I20260928 13:08:00.685410 10190 raft_consensus.cc:2229] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:00.685559 10190 raft_consensus.cc:2258] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:00.686760 10191 tablet_replica.cc:330] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a: stopping tablet replica
I20260928 13:08:00.686901 10191 raft_consensus.cc:2229] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:00.687034 10190 ts_tablet_manager.cc:1905] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:00.687129 10191 raft_consensus.cc:2258] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:00.687453 10192 ts_tablet_manager.cc:1918] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:00.687541 10192 log.cc:1196] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/wal/wals/f9d1b0691803445897da65bad9ef40b8
I20260928 13:08:00.687837 10192 ts_tablet_manager.cc:1939] T f9d1b0691803445897da65bad9ef40b8 P 099fd9dc97514f099771e20f2bc4af1a: Deleting consensus metadata
I20260928 13:08:00.688390 9646 catalog_manager.cc:4841] TS 099fd9dc97514f099771e20f2bc4af1a (127.8.88.129:45173): tablet f9d1b0691803445897da65bad9ef40b8 (table test-table [id=86570d59869d4000b09d725c8cef7281]) successfully deleted
I20260928 13:08:00.688632 10191 ts_tablet_manager.cc:1905] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:00.688858 10190 ts_tablet_manager.cc:1918] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:00.688969 10190 log.cc:1196] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/f9d1b0691803445897da65bad9ef40b8
I20260928 13:08:00.689255 10190 ts_tablet_manager.cc:1939] T f9d1b0691803445897da65bad9ef40b8 P c424a71075a745f883767f26c89805a6: Deleting consensus metadata
I20260928 13:08:00.689761 9647 catalog_manager.cc:4841] TS c424a71075a745f883767f26c89805a6 (127.8.88.130:40121): tablet f9d1b0691803445897da65bad9ef40b8 (table test-table [id=86570d59869d4000b09d725c8cef7281]) successfully deleted
I20260928 13:08:00.690568 10191 ts_tablet_manager.cc:1918] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:00.690657 10191 log.cc:1196] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/f9d1b0691803445897da65bad9ef40b8
I20260928 13:08:00.690699 10169 tablet_bootstrap.cc:492] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6: No bootstrap required, opened a new log
I20260928 13:08:00.690773 10169 ts_tablet_manager.cc:1397] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260928 13:08:00.690934 10191 ts_tablet_manager.cc:1939] T f9d1b0691803445897da65bad9ef40b8 P 26da8b8d723a4fd9af6945816834314a: Deleting consensus metadata
I20260928 13:08:00.691136 10169 raft_consensus.cc:348] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.691229 10169 raft_consensus.cc:374] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:00.691253 10169 raft_consensus.cc:729] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c424a71075a745f883767f26c89805a6, State: Initialized, Role: FOLLOWER
I20260928 13:08:00.691298 10169 consensus_queue.cc:260] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.691376 9647 catalog_manager.cc:4841] TS 26da8b8d723a4fd9af6945816834314a (127.8.88.131:38309): tablet f9d1b0691803445897da65bad9ef40b8 (table test-table [id=86570d59869d4000b09d725c8cef7281]) successfully deleted
I20260928 13:08:00.691452 10169 ts_tablet_manager.cc:1428] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:00.691843 10168 tablet_bootstrap.cc:492] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a: No bootstrap required, opened a new log
I20260928 13:08:00.691911 10168 ts_tablet_manager.cc:1397] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260928 13:08:00.692090 10168 raft_consensus.cc:348] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.692167 10168 raft_consensus.cc:374] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:00.692200 10168 raft_consensus.cc:729] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26da8b8d723a4fd9af6945816834314a, State: Initialized, Role: FOLLOWER
I20260928 13:08:00.692246 10168 consensus_queue.cc:260] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.692354 10168 ts_tablet_manager.cc:1428] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:00.779006 10175 raft_consensus.cc:482] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:00.779131 10175 raft_consensus.cc:504] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.779489 10175 leader_election.cc:288] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 099fd9dc97514f099771e20f2bc4af1a (127.8.88.129:45173), c424a71075a745f883767f26c89805a6 (127.8.88.130:40121)
I20260928 13:08:00.783489 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.779733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:40121 (local address 127.8.88.131:40109)
0928 13:08:00.779815 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.779829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.779876 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.780070 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.780072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.780074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:00.780141 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.780143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.781120 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.781122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.782377 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.782380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.783220 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.783221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.783303 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.783308 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.783356 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.783402 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 13:08:00.783864 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.779832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:40109 (local address 127.8.88.130:40121)
0928 13:08:00.779881 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.779883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.779926 (+ 43us) server_negotiation.cc:407] Connection header received
0928 13:08:00.779955 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.779956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.779968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.780016 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:00.780214 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.781009 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.782505 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.783092 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.783175 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.783185 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.783728 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.783793 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 13:08:00.784053 9958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f2b96f57a9492dbb37258c07c8703a" candidate_uuid: "26da8b8d723a4fd9af6945816834314a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c424a71075a745f883767f26c89805a6" is_pre_election: true
I20260928 13:08:00.784155 9958 raft_consensus.cc:2454] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26da8b8d723a4fd9af6945816834314a in term 0.
I20260928 13:08:00.784299 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.780011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:57111 (local address 127.8.88.129:45173)
0928 13:08:00.780071 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.780073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.780081 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:08:00.780111 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.780112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.780121 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.780178 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:00.780411 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.781199 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.782825 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.783400 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.783453 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.783463 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.784180 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.784232 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 13:08:00.784412 10031 leader_election.cc:302] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26da8b8d723a4fd9af6945816834314a, c424a71075a745f883767f26c89805a6; no voters:
I20260928 13:08:00.784708 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.779733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:45173 (local address 127.8.88.131:57111)
0928 13:08:00.779916 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.779925 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.779972 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.780224 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.780226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.780228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:00.780315 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.780317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.781312 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.781313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.782669 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.782672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.783547 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.783548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.783876 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.783882 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.784580 (+ 698us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.784629 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":16,"thread_start_us":71,"threads_started":1}
I20260928 13:08:00.785091 10175 raft_consensus.cc:2781] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:00.785152 10175 raft_consensus.cc:482] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:00.785174 10175 raft_consensus.cc:3037] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:00.785087 9818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f2b96f57a9492dbb37258c07c8703a" candidate_uuid: "26da8b8d723a4fd9af6945816834314a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "099fd9dc97514f099771e20f2bc4af1a" is_pre_election: true
I20260928 13:08:00.785221 9818 raft_consensus.cc:2454] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26da8b8d723a4fd9af6945816834314a in term 0.
I20260928 13:08:00.785924 10175 raft_consensus.cc:504] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.786108 10175 leader_election.cc:288] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [CANDIDATE]: Term 1 election: Requested vote from peers 099fd9dc97514f099771e20f2bc4af1a (127.8.88.129:45173), c424a71075a745f883767f26c89805a6 (127.8.88.130:40121)
I20260928 13:08:00.786290 9818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f2b96f57a9492dbb37258c07c8703a" candidate_uuid: "26da8b8d723a4fd9af6945816834314a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "099fd9dc97514f099771e20f2bc4af1a"
I20260928 13:08:00.786320 9958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30f2b96f57a9492dbb37258c07c8703a" candidate_uuid: "26da8b8d723a4fd9af6945816834314a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c424a71075a745f883767f26c89805a6"
I20260928 13:08:00.786383 9818 raft_consensus.cc:3037] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:00.786394 9958 raft_consensus.cc:3037] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:00.787215 9958 raft_consensus.cc:2454] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26da8b8d723a4fd9af6945816834314a in term 1.
I20260928 13:08:00.787214 9818 raft_consensus.cc:2454] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26da8b8d723a4fd9af6945816834314a in term 1.
I20260928 13:08:00.787415 10031 leader_election.cc:302] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26da8b8d723a4fd9af6945816834314a, c424a71075a745f883767f26c89805a6; no voters:
I20260928 13:08:00.787531 10175 raft_consensus.cc:2781] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:00.787694 10175 raft_consensus.cc:686] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [term 1 LEADER]: Becoming Leader. State: Replica: 26da8b8d723a4fd9af6945816834314a, State: Running, Role: LEADER
I20260928 13:08:00.787766 10175 consensus_queue.cc:237] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.788576 9660 catalog_manager.cc:5495] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a reported cstate change: term changed from 0 to 1, leader changed from <none> to 26da8b8d723a4fd9af6945816834314a (127.8.88.131). New cstate: current_term: 1 leader_uuid: "26da8b8d723a4fd9af6945816834314a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } health_report { overall_health: HEALTHY } } }
Sep 28 13:08:00 dist-test-slave-p1lc krb5kdc[9616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600879, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:00.829631 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.821742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43189 (local address 127.0.0.1:43732)
0928 13:08:00.822017 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.822031 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.822129 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.822374 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.822375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.822711 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:00.822788 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.822791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.823476 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.823477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.824003 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.824005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.824364 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.824365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.824414 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.825013 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:00.825018 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:00.826634 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:00.828240 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:00.828248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:00.828251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:00.828497 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:00.828733 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:00.828733 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:00.828734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:00.828818 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:00.829334 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:00.829343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:00.829392 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.829455 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.829556 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":144,"threads_started":1}
I20260928 13:08:00.829692 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.821865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.8.88.190:43189)
0928 13:08:00.821994 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.821997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.822108 (+ 111us) server_negotiation.cc:407] Connection header received
0928 13:08:00.822210 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.822211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.822222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.822367 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:00.822878 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.823393 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.824088 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.824288 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.824366 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.826758 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:00.826759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:00.826759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:00.826767 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:00.828079 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:00.828592 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:00.828593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:00.828594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:00.828664 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:00.829064 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:00.829066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:00.829066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:00.829255 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:00.829328 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.829515 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.829606 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":97}
I20260928 13:08:00.832278 9660 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:43732:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 13:08:00.832522 9660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:00.834474 10078 tablet_service.cc:1467] Processing CreateTablet for tablet f392f1022a7c41cba3e7bdd8cac6e08c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f78d562ac5f34cdfac71e36880eb6bbb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 13:08:00.834460 9798 tablet_service.cc:1467] Processing CreateTablet for tablet f392f1022a7c41cba3e7bdd8cac6e08c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f78d562ac5f34cdfac71e36880eb6bbb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 13:08:00.834460 9938 tablet_service.cc:1467] Processing CreateTablet for tablet f392f1022a7c41cba3e7bdd8cac6e08c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f78d562ac5f34cdfac71e36880eb6bbb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 13:08:00.834645 9938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f392f1022a7c41cba3e7bdd8cac6e08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:00.834647 9798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f392f1022a7c41cba3e7bdd8cac6e08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:00.834690 10078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f392f1022a7c41cba3e7bdd8cac6e08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:00.836050 10168 tablet_bootstrap.cc:492] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a: Bootstrap starting.
I20260928 13:08:00.836201 10169 tablet_bootstrap.cc:492] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6: Bootstrap starting.
I20260928 13:08:00.836288 10167 tablet_bootstrap.cc:492] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a: Bootstrap starting.
I20260928 13:08:00.836493 10168 tablet_bootstrap.cc:654] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:00.836596 10167 tablet_bootstrap.cc:654] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:00.836596 10169 tablet_bootstrap.cc:654] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:00.837162 10168 tablet_bootstrap.cc:492] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a: No bootstrap required, opened a new log
I20260928 13:08:00.837216 10168 ts_tablet_manager.cc:1397] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:00.837277 10167 tablet_bootstrap.cc:492] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a: No bootstrap required, opened a new log
I20260928 13:08:00.837301 10169 tablet_bootstrap.cc:492] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6: No bootstrap required, opened a new log
I20260928 13:08:00.837320 10167 ts_tablet_manager.cc:1397] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:00.837337 10169 ts_tablet_manager.cc:1397] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:00.837365 10168 raft_consensus.cc:348] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.837443 10168 raft_consensus.cc:374] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:00.837420 10167 raft_consensus.cc:348] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.837463 10168 raft_consensus.cc:729] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26da8b8d723a4fd9af6945816834314a, State: Initialized, Role: FOLLOWER
I20260928 13:08:00.837447 10169 raft_consensus.cc:348] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.837482 10167 raft_consensus.cc:374] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:00.837507 10167 raft_consensus.cc:729] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099fd9dc97514f099771e20f2bc4af1a, State: Initialized, Role: FOLLOWER
I20260928 13:08:00.837496 10169 raft_consensus.cc:374] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:00.837517 10168 consensus_queue.cc:260] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.837546 10169 raft_consensus.cc:729] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c424a71075a745f883767f26c89805a6, State: Initialized, Role: FOLLOWER
I20260928 13:08:00.837549 10167 consensus_queue.cc:260] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.837597 10169 consensus_queue.cc:260] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.837611 10168 ts_tablet_manager.cc:1428] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:00.837646 10167 ts_tablet_manager.cc:1428] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:00.837685 10169 ts_tablet_manager.cc:1428] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 13:08:00.837716 10173 tablet_replica.cc:439] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } }
I20260928 13:08:00.837710 10175 tablet_replica.cc:439] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } }
I20260928 13:08:00.837790 10175 tablet_replica.cc:442] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:00.837788 10173 tablet_replica.cc:442] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:00.837803 10174 tablet_replica.cc:439] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } }
I20260928 13:08:00.838019 10174 tablet_replica.cc:442] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:00.971673 10174 raft_consensus.cc:482] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:00.971796 10174 raft_consensus.cc:504] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.972185 10174 leader_election.cc:288] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 099fd9dc97514f099771e20f2bc4af1a (127.8.88.129:45173), 26da8b8d723a4fd9af6945816834314a (127.8.88.131:38309)
I20260928 13:08:00.975222 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.972431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:35851 (local address 127.8.88.129:45173)
0928 13:08:00.972514 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.972516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.972584 (+ 68us) server_negotiation.cc:407] Connection header received
0928 13:08:00.972667 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.972668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.972681 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.972762 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:00.972957 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.973460 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.974477 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.974851 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.974941 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.974952 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.975105 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.975141 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 13:08:00.975270 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.972580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:48203 (local address 127.8.88.131:38309)
0928 13:08:00.972624 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.972626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.972737 (+ 111us) server_negotiation.cc:407] Connection header received
0928 13:08:00.972809 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.972811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.972820 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.972915 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:00.973072 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.973604 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.974542 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.974903 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.974988 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.974995 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.975161 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.975201 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:08:00.975291 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.972411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:38309 (local address 127.8.88.130:48203)
0928 13:08:00.972658 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.972673 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.972754 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.972921 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.972922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.972923 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:00.973000 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.973002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.973695 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.973696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.974448 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.974450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.974990 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.974991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.975073 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.975076 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.975143 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.975185 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":122,"threads_started":1}
I20260928 13:08:00.975481 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.972359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:45173 (local address 127.8.88.130:35851)
0928 13:08:00.972508 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.972520 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.972590 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.972782 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.972782 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.972785 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:00.972858 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.972860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.973554 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.973555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.974379 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.974382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.974960 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.974961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.975026 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.975029 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.975098 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.975141 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260928 13:08:00.975541 10098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f392f1022a7c41cba3e7bdd8cac6e08c" candidate_uuid: "c424a71075a745f883767f26c89805a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26da8b8d723a4fd9af6945816834314a" is_pre_election: true
I20260928 13:08:00.975646 10098 raft_consensus.cc:2454] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c424a71075a745f883767f26c89805a6 in term 0.
I20260928 13:08:00.975718 9818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f392f1022a7c41cba3e7bdd8cac6e08c" candidate_uuid: "c424a71075a745f883767f26c89805a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "099fd9dc97514f099771e20f2bc4af1a" is_pre_election: true
I20260928 13:08:00.975821 9818 raft_consensus.cc:2454] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c424a71075a745f883767f26c89805a6 in term 0.
I20260928 13:08:00.975878 9891 leader_election.cc:302] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26da8b8d723a4fd9af6945816834314a, c424a71075a745f883767f26c89805a6; no voters:
I20260928 13:08:00.976049 10174 raft_consensus.cc:2781] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:00.976128 10174 raft_consensus.cc:482] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:00.976167 10174 raft_consensus.cc:3037] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:00.976895 10174 raft_consensus.cc:504] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.977063 10174 leader_election.cc:288] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [CANDIDATE]: Term 1 election: Requested vote from peers 099fd9dc97514f099771e20f2bc4af1a (127.8.88.129:45173), 26da8b8d723a4fd9af6945816834314a (127.8.88.131:38309)
I20260928 13:08:00.977233 9818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f392f1022a7c41cba3e7bdd8cac6e08c" candidate_uuid: "c424a71075a745f883767f26c89805a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "099fd9dc97514f099771e20f2bc4af1a"
I20260928 13:08:00.977284 10098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f392f1022a7c41cba3e7bdd8cac6e08c" candidate_uuid: "c424a71075a745f883767f26c89805a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26da8b8d723a4fd9af6945816834314a"
I20260928 13:08:00.977366 10098 raft_consensus.cc:3037] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:00.977366 9818 raft_consensus.cc:3037] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:00.978121 10098 raft_consensus.cc:2454] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c424a71075a745f883767f26c89805a6 in term 1.
I20260928 13:08:00.978121 9818 raft_consensus.cc:2454] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c424a71075a745f883767f26c89805a6 in term 1.
I20260928 13:08:00.978340 9891 leader_election.cc:302] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26da8b8d723a4fd9af6945816834314a, c424a71075a745f883767f26c89805a6; no voters:
I20260928 13:08:00.978473 10174 raft_consensus.cc:2781] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:00.978646 10174 raft_consensus.cc:686] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [term 1 LEADER]: Becoming Leader. State: Replica: c424a71075a745f883767f26c89805a6, State: Running, Role: LEADER
I20260928 13:08:00.978734 10174 consensus_queue.cc:237] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } }
I20260928 13:08:00.979125 10210 tablet_replica.cc:439] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6: TxnStatusTablet state changed. Reason: New leader c424a71075a745f883767f26c89805a6. Latest consensus state: current_term: 1 leader_uuid: "c424a71075a745f883767f26c89805a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } }
I20260928 13:08:00.979241 10210 tablet_replica.cc:442] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6: This TxnStatusTablet replica's current role is: LEADER
I20260928 13:08:00.979409 10212 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 13:08:00.979535 10212 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 13:08:00.979525 9660 catalog_manager.cc:5495] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to c424a71075a745f883767f26c89805a6 (127.8.88.130). New cstate: current_term: 1 leader_uuid: "c424a71075a745f883767f26c89805a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } health_report { overall_health: UNKNOWN } } }
I20260928 13:08:00.989784 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.986780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.8.88.130:40121)
0928 13:08:00.986854 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.986856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.986864 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:08:00.986898 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.986899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.986918 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.986988 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:00.987159 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.987905 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.988754 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.989097 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.989181 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.989429 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:00.989557 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:00.989632 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.989682 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.989703 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 13:08:00.989782 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.986485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:40121 (local address 127.0.0.1:57574)
0928 13:08:00.986567 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:00.986579 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:00.986628 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:00.987015 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:00.987016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:00.987018 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 13:08:00.987074 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.987077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.987982 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.987983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.988662 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:00.988665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.989192 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.989194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:00.989310 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.989329 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 13:08:00.989626 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 13:08:00.989632 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 13:08:00.989677 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 13:08:00.989720 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 13:08:00.991025 10098 raft_consensus.cc:1261] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a [term 1 FOLLOWER]: Refusing update from remote peer c424a71075a745f883767f26c89805a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:00.991179 9818 raft_consensus.cc:1261] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a [term 1 FOLLOWER]: Refusing update from remote peer c424a71075a745f883767f26c89805a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:00.991405 10174 consensus_queue.cc:1035] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:00.991552 10210 consensus_queue.cc:1035] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:00.992053 10175 tablet_replica.cc:439] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a: TxnStatusTablet state changed. Reason: New leader c424a71075a745f883767f26c89805a6. Latest consensus state: current_term: 1 leader_uuid: "c424a71075a745f883767f26c89805a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } }
I20260928 13:08:00.992142 10175 tablet_replica.cc:442] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:00.992100 10173 tablet_replica.cc:439] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a: TxnStatusTablet state changed. Reason: New leader c424a71075a745f883767f26c89805a6. Latest consensus state: current_term: 1 leader_uuid: "c424a71075a745f883767f26c89805a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } }
I20260928 13:08:00.992197 10173 tablet_replica.cc:442] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:00.992974 10210 tablet_replica.cc:439] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c424a71075a745f883767f26c89805a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } }
I20260928 13:08:00.993062 10210 tablet_replica.cc:442] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6: This TxnStatusTablet replica's current role is: LEADER
I20260928 13:08:00.993115 10173 tablet_replica.cc:439] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c424a71075a745f883767f26c89805a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } }
I20260928 13:08:00.993206 10173 tablet_replica.cc:442] T f392f1022a7c41cba3e7bdd8cac6e08c P 099fd9dc97514f099771e20f2bc4af1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:00.994282 10174 tablet_replica.cc:439] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c424a71075a745f883767f26c89805a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } }
I20260928 13:08:00.994385 10174 tablet_replica.cc:442] T f392f1022a7c41cba3e7bdd8cac6e08c P c424a71075a745f883767f26c89805a6: This TxnStatusTablet replica's current role is: LEADER
I20260928 13:08:00.994661 10175 tablet_replica.cc:439] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c424a71075a745f883767f26c89805a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 } } peers { permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 } } peers { permanent_uuid: "26da8b8d723a4fd9af6945816834314a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 38309 } } }
I20260928 13:08:00.994760 10175 tablet_replica.cc:442] T f392f1022a7c41cba3e7bdd8cac6e08c P 26da8b8d723a4fd9af6945816834314a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:00.997474 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:00.994890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.8.88.131:38309)
0928 13:08:00.994931 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:00.994934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:00.995053 (+ 119us) server_negotiation.cc:407] Connection header received
0928 13:08:00.995085 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:00.995086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:00.995102 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:00.995190 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:00.995373 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.995862 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.996603 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:00.996854 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:00.996972 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:00.997129 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:00.997245 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:00.997325 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:00.997371 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 13:08:00.997392 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:08:01.004261 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:01.001177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:40121 (local address 127.8.88.131:55583)
0928 13:08:01.001386 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:01.001424 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:01.001497 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:01.001641 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:01.001641 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:01.001642 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 13:08:01.001726 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:01.001728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:01.002719 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:01.002720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:01.003402 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:01.003405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:01.003886 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:01.003887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:01.003937 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:01.003952 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 13:08:01.004120 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 13:08:01.004121 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 13:08:01.004156 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 13:08:01.004190 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260928 13:08:01.004264 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:01.001436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:55583 (local address 127.8.88.130:40121)
0928 13:08:01.001481 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:01.001483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:01.001493 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:08:01.001567 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:01.001568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:01.001581 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:01.001640 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:01.001813 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:01.002635 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:01.003495 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:01.003800 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:01.003874 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:01.004030 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:01.004064 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:01.004120 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:01.004161 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 13:08:01.004182 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:08:01.007114 9958 raft_consensus.cc:1261] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6 [term 1 FOLLOWER]: Refusing update from remote peer 26da8b8d723a4fd9af6945816834314a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:01.007470 10175 consensus_queue.cc:1035] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c424a71075a745f883767f26c89805a6" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:01.007531 9818 raft_consensus.cc:1261] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a [term 1 FOLLOWER]: Refusing update from remote peer 26da8b8d723a4fd9af6945816834314a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:01.007779 10175 consensus_queue.cc:1035] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [LEADER]: Connected to new peer: Peer: permanent_uuid: "099fd9dc97514f099771e20f2bc4af1a" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:01.017520 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:01.014512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:36891 (local address 127.8.88.131:38309)
0928 13:08:01.014601 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:01.014603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:01.014764 (+ 161us) server_negotiation.cc:407] Connection header received
0928 13:08:01.014833 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:01.014834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:01.014843 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:01.014885 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:01.015022 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:01.015482 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:01.016788 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:01.017054 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:01.017110 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:01.017316 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:01.017348 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:01.017384 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:01.017442 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 13:08:01.017459 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 13:08:01.017604 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:01.014472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:38309 (local address 127.8.88.130:36891)
0928 13:08:01.014706 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:01.014718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:01.014776 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:01.014899 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:01.014900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:01.014902 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 13:08:01.014968 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:01.014970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:01.015554 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:01.015555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:01.016663 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:01.016666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:01.017140 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:01.017141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:01.017211 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:01.017231 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 13:08:01.017388 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 13:08:01.017389 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 13:08:01.017439 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 13:08:01.017486 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":136,"threads_started":1}
I20260928 13:08:01.029752 9661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:01.029881 9661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:01.030675 9661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76889e4fd79a4cd39254b6c8ab98258a: Sending DeleteTablet for 3 replicas of tablet 30f2b96f57a9492dbb37258c07c8703a
I20260928 13:08:01.031026 10078 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f2b96f57a9492dbb37258c07c8703a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:01 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:60134
I20260928 13:08:01.031059 9938 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f2b96f57a9492dbb37258c07c8703a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:01 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:57568
I20260928 13:08:01.031131 10191 tablet_replica.cc:330] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a: stopping tablet replica
I20260928 13:08:01.031131 9798 tablet_service.cc:1514] Processing DeleteTablet for tablet 30f2b96f57a9492dbb37258c07c8703a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:01 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:54626
I20260928 13:08:01.031183 10191 raft_consensus.cc:2229] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:08:01.031240 10190 tablet_replica.cc:330] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6: stopping tablet replica
I20260928 13:08:01.031284 10191 raft_consensus.cc:2258] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:01.031296 10190 raft_consensus.cc:2229] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:01.031337 10190 raft_consensus.cc:2258] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:01.031414 10192 tablet_replica.cc:330] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a: stopping tablet replica
I20260928 13:08:01.031471 10192 raft_consensus.cc:2229] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:01.031510 10192 raft_consensus.cc:2258] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:01.032487 10192 ts_tablet_manager.cc:1905] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:01.033138 10190 ts_tablet_manager.cc:1905] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:01.034747 10192 ts_tablet_manager.cc:1918] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:01.034782 10190 ts_tablet_manager.cc:1918] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:01.034816 10192 log.cc:1196] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-0/wal/wals/30f2b96f57a9492dbb37258c07c8703a
I20260928 13:08:01.034842 10190 log.cc:1196] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/30f2b96f57a9492dbb37258c07c8703a
I20260928 13:08:01.035068 10190 ts_tablet_manager.cc:1939] T 30f2b96f57a9492dbb37258c07c8703a P c424a71075a745f883767f26c89805a6: Deleting consensus metadata
I20260928 13:08:01.035073 10192 ts_tablet_manager.cc:1939] T 30f2b96f57a9492dbb37258c07c8703a P 099fd9dc97514f099771e20f2bc4af1a: Deleting consensus metadata
I20260928 13:08:01.035356 10191 ts_tablet_manager.cc:1905] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:01.035539 9646 catalog_manager.cc:4841] TS 099fd9dc97514f099771e20f2bc4af1a (127.8.88.129:45173): tablet 30f2b96f57a9492dbb37258c07c8703a (table test-table [id=eed41887ab32440d8e37fa01c38c570b]) successfully deleted
I20260928 13:08:01.035916 9647 catalog_manager.cc:4841] TS c424a71075a745f883767f26c89805a6 (127.8.88.130:40121): tablet 30f2b96f57a9492dbb37258c07c8703a (table test-table [id=eed41887ab32440d8e37fa01c38c570b]) successfully deleted
I20260928 13:08:01.037714 10191 ts_tablet_manager.cc:1918] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:01.037783 10191 log.cc:1196] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/30f2b96f57a9492dbb37258c07c8703a
I20260928 13:08:01.038010 10191 ts_tablet_manager.cc:1939] T 30f2b96f57a9492dbb37258c07c8703a P 26da8b8d723a4fd9af6945816834314a: Deleting consensus metadata
I20260928 13:08:01.038372 9647 catalog_manager.cc:4841] TS 26da8b8d723a4fd9af6945816834314a (127.8.88.131:38309): tablet 30f2b96f57a9492dbb37258c07c8703a (table test-table [id=eed41887ab32440d8e37fa01c38c570b]) successfully deleted
Sep 28 13:08:01 dist-test-slave-p1lc krb5kdc[9616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600880, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.88.129@KRBTEST.COM
I20260928 13:08:01.050331 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:01.037836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.8.88.129:45173)
0928 13:08:01.037887 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:01.037890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:01.038220 (+ 330us) server_negotiation.cc:407] Connection header received
0928 13:08:01.038343 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:01.038344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:01.038355 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:01.038389 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:01.039273 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:01.040055 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:01.041252 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:01.041541 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:01.041586 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:01.046284 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:01.046301 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:01.046301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:01.046311 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:01.048681 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:01.049241 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:01.049246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:01.049247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:01.049323 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:01.049561 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:01.049562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:01.049563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:01.049766 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:01.049929 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:01.050012 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 13:08:01.050102 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260928 13:08:01.050595 9858 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54642
I20260928 13:08:01.060271 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:01.051657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.8.88.190:43189)
0928 13:08:01.051712 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:01.051715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:01.051977 (+ 262us) server_negotiation.cc:407] Connection header received
0928 13:08:01.052032 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:01.052033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:01.052045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:01.052105 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:01.052754 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:01.053580 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:01.054638 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:01.054960 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:01.055038 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:01.056680 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:01.056681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:01.056682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:01.056690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:01.058916 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:01.059381 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:01.059382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:01.059383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:01.059455 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:01.059682 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:01.059683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:01.059683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:01.059834 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:01.059939 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:01.060082 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 13:08:01.060166 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 13:08:01.060596 9661 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43742
I20260928 13:08:01.061420 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 9733
I20260928 13:08:01.071205 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 9873
I20260928 13:08:01.077732 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 10013
I20260928 13:08:01.083184 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 9629
2026-09-28T13:08:01Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2624 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:08:01 dist-test-slave-p1lc krb5kdc[10235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:08:01 dist-test-slave-p1lc krb5kdc[10235](info): set up 2 sockets
Sep 28 13:08:01 dist-test-slave-p1lc krb5kdc[10235](info): commencing operation
krb5kdc: starting...
W20260928 13:08:02.140996 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:08:02 dist-test-slave-p1lc krb5kdc[10235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:08:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:08:02.268723 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:41029
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:41029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:02.351804 10248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:02.352031 10248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:02.352056 10248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:02.353226 10248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:02.353269 10248 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 13:08:02.353284 10248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:02.353297 10248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:02.353310 10248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:02.353322 10248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:02.354770 10248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:41029
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:41029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:02.354972 10248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:02.355149 10248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:02.360376 10256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:02.360615 10248 server_base.cc:1034] running on GCE node
W20260928 13:08:02.360526 10253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:02.360378 10254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:02.361048 10248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:02.361433 10248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:02.362623 10248 hybrid_clock.cc:648] HybridClock initialized: now 1790600882362663 us; error 91 us; skew 500 ppm
Sep 28 13:08:02 dist-test-slave-p1lc krb5kdc[10235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600882, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:02.365643 10248 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:08:02.366271 10248 webserver.cc:460] Webserver started at http://127.8.88.190:46557/ using document root <none> and password file <none>
I20260928 13:08:02.366534 10248 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:02.366605 10248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:02.366765 10248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:02.367875 10248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "64afc977eba94b05a2344b4de69f0465"
format_stamp: "Formatted at 2026-09-28 13:08:02 on dist-test-slave-p1lc"
server_key: "b1337b18daee9aa69696b1fb768b5527"
server_key_iv: "c7e8372953e3e76085871f6065440a82"
server_key_version: "encryptionkey@0"
I20260928 13:08:02.368309 10248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "64afc977eba94b05a2344b4de69f0465"
format_stamp: "Formatted at 2026-09-28 13:08:02 on dist-test-slave-p1lc"
server_key: "b1337b18daee9aa69696b1fb768b5527"
server_key_iv: "c7e8372953e3e76085871f6065440a82"
server_key_version: "encryptionkey@0"
I20260928 13:08:02.369957 10248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:08:02.371047 10263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:02.371373 10248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:02.371495 10248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "64afc977eba94b05a2344b4de69f0465"
format_stamp: "Formatted at 2026-09-28 13:08:02 on dist-test-slave-p1lc"
server_key: "b1337b18daee9aa69696b1fb768b5527"
server_key_iv: "c7e8372953e3e76085871f6065440a82"
server_key_version: "encryptionkey@0"
I20260928 13:08:02.371585 10248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:02.380416 10248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:02.382902 10248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:02.383080 10248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:02.388036 10248 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:41029
I20260928 13:08:02.388082 10325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:41029 every 8 connection(s)
I20260928 13:08:02.388448 10248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:02.389164 10326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:02.391664 10326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465: Bootstrap starting.
I20260928 13:08:02.392273 10326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:02.392565 10326 log.cc:824] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:02.393414 10326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465: No bootstrap required, opened a new log
I20260928 13:08:02.393925 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 10248
I20260928 13:08:02.394003 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:02.394194 8546 external_mini_cluster.cc:1428] Setting key 9b195132f0c4b08cbcbc9bd15ca17f0d
I20260928 13:08:02.395362 10326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64afc977eba94b05a2344b4de69f0465" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 41029 } }
I20260928 13:08:02.395535 10326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:02.395584 10326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64afc977eba94b05a2344b4de69f0465, State: Initialized, Role: FOLLOWER
I20260928 13:08:02.395747 10326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64afc977eba94b05a2344b4de69f0465" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 41029 } }
I20260928 13:08:02.395846 10326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:02.395900 10326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:02.395954 10326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:02.396517 10326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64afc977eba94b05a2344b4de69f0465" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 41029 } }
I20260928 13:08:02.396641 10326 leader_election.cc:302] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64afc977eba94b05a2344b4de69f0465; no voters:
I20260928 13:08:02.396855 10326 leader_election.cc:288] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:02.396950 10331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:02.397119 10331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [term 1 LEADER]: Becoming Leader. State: Replica: 64afc977eba94b05a2344b4de69f0465, State: Running, Role: LEADER
I20260928 13:08:02.397166 10326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:02.397307 10331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64afc977eba94b05a2344b4de69f0465" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 41029 } }
I20260928 13:08:02.398012 10333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64afc977eba94b05a2344b4de69f0465. Latest consensus state: current_term: 1 leader_uuid: "64afc977eba94b05a2344b4de69f0465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64afc977eba94b05a2344b4de69f0465" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 41029 } } }
I20260928 13:08:02.398133 10333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:02.398348 10340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:02.397998 10332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64afc977eba94b05a2344b4de69f0465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64afc977eba94b05a2344b4de69f0465" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 41029 } } }
I20260928 13:08:02.398713 10332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:02.398986 10340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:02.401018 10340 catalog_manager.cc:1347] Generated new cluster ID: 495835766771410d93b69b5a3346725d
I20260928 13:08:02.401070 10340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:08:02 dist-test-slave-p1lc krb5kdc[10235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:02.405993 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:02.394829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.8.88.190:41029)
0928 13:08:02.395332 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:02.395340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:02.395371 (+ 31us) server_negotiation.cc:407] Connection header received
0928 13:08:02.395495 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:02.395501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:02.395545 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:02.395712 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:02.396446 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.397828 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.399286 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.399834 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.400274 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:02.402588 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:02.402592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:02.402592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:02.402605 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:02.404358 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:02.404708 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:02.404711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:02.404711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:02.404828 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:02.405072 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:02.405073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:02.405073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:02.405313 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:02.405471 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:02.405579 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 13:08:02.405828 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260928 13:08:02.436167 10340 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:02.436861 10340 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:02.445030 10340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465: Generated new TSK 0
I20260928 13:08:02.445344 10340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:08:02.481333 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:41029
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:02.591260 10352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:02.591517 10352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:02.591549 10352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:02.593042 10352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:02.593101 10352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:02.593127 10352 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 13:08:02.593142 10352 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 13:08:02.593163 10352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:08:02.595005 10352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.88.190:41029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:02.595337 10352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:02.595635 10352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:02.601428 10360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:02.601552 10357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:02.601574 10358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:02.601905 10352 server_base.cc:1034] running on GCE node
I20260928 13:08:02.602102 10352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:02.602314 10352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:02.603495 10352 hybrid_clock.cc:648] HybridClock initialized: now 1790600882603472 us; error 40 us; skew 500 ppm
Sep 28 13:08:02 dist-test-slave-p1lc krb5kdc[10235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600882, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:02.606179 10352 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:08:02.606751 10352 webserver.cc:460] Webserver started at http://127.8.88.129:39369/ using document root <none> and password file <none>
I20260928 13:08:02.607003 10352 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:02.607075 10352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:02.607226 10352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:02.608430 10352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "dea8521539894639bb46cb94342e70fd"
format_stamp: "Formatted at 2026-09-28 13:08:02 on dist-test-slave-p1lc"
server_key: "558fa7a809af911ea030a3b118cf7abc"
server_key_iv: "00fa794611b2152ead4e7af314fc2e32"
server_key_version: "encryptionkey@0"
I20260928 13:08:02.608860 10352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "dea8521539894639bb46cb94342e70fd"
format_stamp: "Formatted at 2026-09-28 13:08:02 on dist-test-slave-p1lc"
server_key: "558fa7a809af911ea030a3b118cf7abc"
server_key_iv: "00fa794611b2152ead4e7af314fc2e32"
server_key_version: "encryptionkey@0"
I20260928 13:08:02.610430 10352 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:02.611433 10367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:02.611794 10352 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:02.611928 10352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "dea8521539894639bb46cb94342e70fd"
format_stamp: "Formatted at 2026-09-28 13:08:02 on dist-test-slave-p1lc"
server_key: "558fa7a809af911ea030a3b118cf7abc"
server_key_iv: "00fa794611b2152ead4e7af314fc2e32"
server_key_version: "encryptionkey@0"
I20260928 13:08:02.612030 10352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:02.625341 10352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:02.628703 10352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:02.628976 10352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:02.629660 10352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:02.629726 10352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:02.629761 10352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:02.629782 10352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 13:08:02 dist-test-slave-p1lc krb5kdc[10235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600882, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:02.638846 10352 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:44391
I20260928 13:08:02.639240 10352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:08:02.639763 10482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:44391 every 8 connection(s)
I20260928 13:08:02.642902 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:02.630150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:35145 (local address 127.8.88.190:41029)
0928 13:08:02.630249 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:02.630251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:02.630556 (+ 305us) server_negotiation.cc:407] Connection header received
0928 13:08:02.631352 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:02.631354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:02.631372 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:02.631441 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:02.632590 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.633354 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.635586 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.635843 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.635953 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:02.639369 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:02.639371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:02.639372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:02.639382 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:02.641226 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:02.641759 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:02.641760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:02.641761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:02.641838 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:02.642141 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:02.642142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:02.642142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:02.642284 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:02.642369 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:02.642685 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 13:08:02.642798 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 13:08:02.643131 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:02.630155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:41029 (local address 127.8.88.129:35145)
0928 13:08:02.630373 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:02.630434 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:02.631193 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:02.631552 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:02.631589 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:02.632055 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:02.632435 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:02.632443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.633463 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.633464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:02.635433 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:02.635439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.635981 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.635983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:02.636196 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:02.637166 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:02.637175 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:02.639221 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:02.641391 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:02.641400 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:02.641402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:02.641666 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:02.641925 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:02.641926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:02.641927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:02.642056 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:02.642360 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:02.642368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:02.642533 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 13:08:02.642660 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 13:08:02.642856 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":109,"threads_started":1}
I20260928 13:08:02.643702 10483 heartbeater.cc:347] Connected to a master server at 127.8.88.190:41029
I20260928 13:08:02.643798 10483 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:02.644021 10483 heartbeater.cc:511] Master 127.8.88.190:41029 requested a full tablet report, sending...
I20260928 13:08:02.644667 10279 ts_manager.cc:194] Registered new tserver with Master: dea8521539894639bb46cb94342e70fd (127.8.88.129:44391)
I20260928 13:08:02.645658 10279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:35145
I20260928 13:08:02.647993 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 10352
I20260928 13:08:02.648120 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:08:02.648361 8546 external_mini_cluster.cc:1428] Setting key 7fa58d822385bb348a1a899b32e55096
I20260928 13:08:02.651539 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:02.645127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:35767 (local address 127.8.88.190:41029)
0928 13:08:02.645174 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:02.645176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:02.645257 (+ 81us) server_negotiation.cc:407] Connection header received
0928 13:08:02.645311 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:02.645312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:02.645321 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:02.645365 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:02.645751 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.646613 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.647330 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.647540 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.647600 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:02.648809 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:02.648810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:02.648811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:02.648819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:02.650316 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:02.650704 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:02.650705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:02.650706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:02.650765 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:02.650983 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:02.650984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:02.650984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:02.651102 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:02.651173 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:02.651363 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 13:08:02.651443 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:08:02.651552 10489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:02.645027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:41029 (local address 127.8.88.129:35767)
0928 13:08:02.645186 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:02.645198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:02.645264 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:02.645381 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:02.645382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:02.645571 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:02.645653 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:02.645655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.646719 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.646720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:02.647247 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:02.647248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.647618 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.647619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:02.647678 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:02.648251 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:02.648256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:02.648707 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:02.650423 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:02.650424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:02.650425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:02.650612 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:02.650849 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:02.650850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:02.650850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:02.650913 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:02.651175 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:02.651177 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:02.651239 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 13:08:02.651325 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 13:08:02.651445 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:08:02.677057 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:41029
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:02.760389 10492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:02.760617 10492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:02.760645 10492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:02.761929 10492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:02.761982 10492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:02.762004 10492 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 13:08:02.762017 10492 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 13:08:02.762035 10492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:08:02.763573 10492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.88.190:41029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:02.763839 10492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:02.764102 10492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:02.769654 10497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:02.769698 10498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:02.769686 10500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:02.769999 10492 server_base.cc:1034] running on GCE node
I20260928 13:08:02.770279 10492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:02.770551 10492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:02.771713 10492 hybrid_clock.cc:648] HybridClock initialized: now 1790600882771695 us; error 37 us; skew 500 ppm
Sep 28 13:08:02 dist-test-slave-p1lc krb5kdc[10235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600882, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:02.775018 10492 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:08:02.775682 10492 webserver.cc:460] Webserver started at http://127.8.88.130:43711/ using document root <none> and password file <none>
I20260928 13:08:02.775959 10492 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:02.776038 10492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:02.776274 10492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:02.777570 10492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "e40d772ccaa14d49b4dc537318a75bcb"
format_stamp: "Formatted at 2026-09-28 13:08:02 on dist-test-slave-p1lc"
server_key: "8cc6d686db9aa18ff1848f341caedb8b"
server_key_iv: "863b1d1bfa8018cf28f501abae8dbe1f"
server_key_version: "encryptionkey@0"
I20260928 13:08:02.777999 10492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "e40d772ccaa14d49b4dc537318a75bcb"
format_stamp: "Formatted at 2026-09-28 13:08:02 on dist-test-slave-p1lc"
server_key: "8cc6d686db9aa18ff1848f341caedb8b"
server_key_iv: "863b1d1bfa8018cf28f501abae8dbe1f"
server_key_version: "encryptionkey@0"
I20260928 13:08:02.779639 10492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:08:02.780715 10507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:02.781025 10492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:02.781246 10492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "e40d772ccaa14d49b4dc537318a75bcb"
format_stamp: "Formatted at 2026-09-28 13:08:02 on dist-test-slave-p1lc"
server_key: "8cc6d686db9aa18ff1848f341caedb8b"
server_key_iv: "863b1d1bfa8018cf28f501abae8dbe1f"
server_key_version: "encryptionkey@0"
I20260928 13:08:02.781375 10492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:02.799960 10492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:02.802460 10492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:02.802671 10492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:02.803438 10492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:02.803509 10492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:02.803550 10492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:02.803603 10492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 13:08:02 dist-test-slave-p1lc krb5kdc[10235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600882, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:02.811935 10492 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:44859
I20260928 13:08:02.812062 10622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:44859 every 8 connection(s)
I20260928 13:08:02.812384 10492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:08:02.813011 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 10492
I20260928 13:08:02.813112 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:08:02.813302 8546 external_mini_cluster.cc:1428] Setting key a6ecfcacf1b08ba5dbaea51e3684f1a1
I20260928 13:08:02.815599 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:02.803908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:38055 (local address 127.8.88.190:41029)
0928 13:08:02.803986 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:02.803989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:02.804255 (+ 266us) server_negotiation.cc:407] Connection header received
0928 13:08:02.804783 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:02.804785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:02.804800 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:02.804910 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:02.805723 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.806233 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.807473 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.807720 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.807811 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:02.811843 (+ 4032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:02.811845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:02.811845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:02.811855 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:02.813767 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:02.814362 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:02.814363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:02.814364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:02.814431 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:02.814694 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:02.814694 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:02.814695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:02.814858 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:02.814929 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:02.815365 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 13:08:02.815462 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 13:08:02.815721 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:02.803901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:41029 (local address 127.8.88.130:38055)
0928 13:08:02.804092 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:02.804134 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:02.804656 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:02.805021 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:02.805025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:02.805300 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:02.805607 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:02.805613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.806336 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.806337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:02.807377 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:02.807380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.807822 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.807824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:02.807986 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:02.808635 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:02.808684 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:02.811647 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:02.813855 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:02.813862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:02.813865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:02.814204 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:02.814520 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:02.814521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:02.814521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:02.814623 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:02.814975 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:02.814982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:02.815151 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 13:08:02.815316 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 13:08:02.815506 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":122,"threads_started":1}
I20260928 13:08:02.816201 10623 heartbeater.cc:347] Connected to a master server at 127.8.88.190:41029
I20260928 13:08:02.816304 10623 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:02.816519 10623 heartbeater.cc:511] Master 127.8.88.190:41029 requested a full tablet report, sending...
I20260928 13:08:02.817005 10279 ts_manager.cc:194] Registered new tserver with Master: e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130:44859)
I20260928 13:08:02.817448 10279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:38055
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
I20260928 13:08:02.824478 10630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:02.817440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:41029 (local address 127.8.88.130:38903)
0928 13:08:02.817594 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:02.817607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:02.817671 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:02.817821 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:02.817823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:02.818054 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:02.818116 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:02.818118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.819176 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.819177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:02.819807 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:02.819809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.820356 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.820357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:02.820423 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:02.820985 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:02.820990 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:02.821310 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:02.823155 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:02.823156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:02.823157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:02.823393 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:02.823611 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:02.823612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:02.823612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:02.823669 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:02.824074 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:02.824075 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:02.824127 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 13:08:02.824197 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 13:08:02.824404 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260928 13:08:02.824491 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:02.817392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:38903 (local address 127.8.88.190:41029)
0928 13:08:02.817460 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:02.817462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:02.817663 (+ 201us) server_negotiation.cc:407] Connection header received
0928 13:08:02.817724 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:02.817725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:02.817737 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:02.817803 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:02.818252 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.819074 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.819909 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.820255 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.820341 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:02.821409 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:02.821411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:02.821411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:02.821418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:02.823048 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:02.823482 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:02.823483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:02.823484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:02.823543 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:02.823779 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:02.823781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:02.823781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:02.823974 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:02.824038 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:02.824276 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 13:08:02.824386 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:08:02.845863 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:41029
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:02.939441 10632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:02.939738 10632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:02.939783 10632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:02.941198 10632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:02.941277 10632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:02.941314 10632 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 13:08:02.941344 10632 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 13:08:02.941378 10632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:08:02.943043 10632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.88.190:41029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:02.943397 10632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:02.943698 10632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:02.949333 10638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:02.949338 10637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:02.949397 10632 server_base.cc:1034] running on GCE node
W20260928 13:08:02.949362 10640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:02.949734 10632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:02.950009 10632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:02.951189 10632 hybrid_clock.cc:648] HybridClock initialized: now 1790600882951171 us; error 29 us; skew 500 ppm
Sep 28 13:08:02 dist-test-slave-p1lc krb5kdc[10235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600882, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:02.954188 10632 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:08:02.954672 10632 webserver.cc:460] Webserver started at http://127.8.88.131:43109/ using document root <none> and password file <none>
I20260928 13:08:02.954926 10632 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:02.954998 10632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:02.955153 10632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:02.956254 10632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "567f476870114c22b5315ab397de9eb2"
format_stamp: "Formatted at 2026-09-28 13:08:02 on dist-test-slave-p1lc"
server_key: "6f91bb1a71a7f341e367c073790ffd39"
server_key_iv: "158af9dfafd2d3b90a97174ba04f173d"
server_key_version: "encryptionkey@0"
I20260928 13:08:02.956718 10632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "567f476870114c22b5315ab397de9eb2"
format_stamp: "Formatted at 2026-09-28 13:08:02 on dist-test-slave-p1lc"
server_key: "6f91bb1a71a7f341e367c073790ffd39"
server_key_iv: "158af9dfafd2d3b90a97174ba04f173d"
server_key_version: "encryptionkey@0"
I20260928 13:08:02.958248 10632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:02.959160 10647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:02.959437 10632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:02.959545 10632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "567f476870114c22b5315ab397de9eb2"
format_stamp: "Formatted at 2026-09-28 13:08:02 on dist-test-slave-p1lc"
server_key: "6f91bb1a71a7f341e367c073790ffd39"
server_key_iv: "158af9dfafd2d3b90a97174ba04f173d"
server_key_version: "encryptionkey@0"
I20260928 13:08:02.959627 10632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:02.978291 10632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:02.980789 10632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:02.981006 10632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:02.981612 10632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:02.981683 10632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:02.981729 10632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:02.981766 10632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:02.988642 10632 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:35645
I20260928 13:08:02.988811 10762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:35645 every 8 connection(s)
I20260928 13:08:02.989099 10632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
Sep 28 13:08:02 dist-test-slave-p1lc krb5kdc[10235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600882, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:02.991832 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 10632
I20260928 13:08:02.991935 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:08:02.992120 8546 external_mini_cluster.cc:1428] Setting key 45bb91305b8dd96bc94dea595325d713
I20260928 13:08:02.993352 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:02.982301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:43379 (local address 127.8.88.190:41029)
0928 13:08:02.982392 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:02.982395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:02.982552 (+ 157us) server_negotiation.cc:407] Connection header received
0928 13:08:02.983160 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:02.983161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:02.983178 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:02.983256 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:02.984381 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.985363 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.987039 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.987391 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.987483 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:02.990281 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:02.990283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:02.990283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:02.990291 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:02.991660 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:02.992201 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:02.992202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:02.992203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:02.992270 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:02.992590 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:02.992591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:02.992592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:02.992723 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:02.992816 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:02.993173 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 13:08:02.993262 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 13:08:02.993651 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:02.982193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:41029 (local address 127.8.88.131:43379)
0928 13:08:02.982398 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:02.982452 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:02.982996 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:02.983363 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:02.983367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:02.983813 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:02.984253 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:02.984261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.985486 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.985487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:02.986931 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:02.986934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.987492 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.987493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:02.987659 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:02.988335 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:02.988356 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:02.990159 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:02.991748 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:02.991757 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:02.991761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:02.992104 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:02.992363 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:02.992363 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:02.992364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:02.992522 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:02.992809 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:02.992815 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:02.993029 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 13:08:02.993149 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 13:08:02.993409 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
I20260928 13:08:02.994247 10763 heartbeater.cc:347] Connected to a master server at 127.8.88.190:41029
I20260928 13:08:02.994354 10763 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:02.994539 10763 heartbeater.cc:511] Master 127.8.88.190:41029 requested a full tablet report, sending...
I20260928 13:08:02.994959 10280 ts_manager.cc:194] Registered new tserver with Master: 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645)
I20260928 13:08:02.995623 10280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:43379
I20260928 13:08:03.001796 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:02.995337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:58735 (local address 127.8.88.190:41029)
0928 13:08:02.995385 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:02.995387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:02.995551 (+ 164us) server_negotiation.cc:407] Connection header received
0928 13:08:02.995578 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:02.995579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:02.995589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:02.995653 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:02.996070 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.996530 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.997433 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.997656 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.997712 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:02.999002 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:02.999003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:02.999004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:02.999010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:03.000611 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:03.001088 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:03.001089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:03.001089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:03.001145 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:03.001332 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:03.001333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:03.001333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:03.001440 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:03.001492 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.001662 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.001723 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:08:03.001819 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:02.995282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:41029 (local address 127.8.88.131:58735)
0928 13:08:02.995479 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:02.995489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:02.995542 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:02.995655 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:02.995656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:02.995900 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:02.995989 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:02.995992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.996622 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.996623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:02.997286 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:02.997288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:02.997774 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:02.997775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:02.997869 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:02.998508 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:02.998514 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:02.998890 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:03.000710 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:03.000711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:03.000712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:03.001014 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:03.001215 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:03.001216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:03.001216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:03.001276 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:03.001525 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:03.001526 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:03.001573 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 13:08:03.001645 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 13:08:03.001738 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":143,"threads_started":1}
I20260928 13:08:03.003957 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 13:08:03 dist-test-slave-p1lc krb5kdc[10235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 13:08:03 dist-test-slave-p1lc krb5kdc[10235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:03.031741 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.023339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.8.88.190:41029)
0928 13:08:03.023416 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.023419 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.023462 (+ 43us) server_negotiation.cc:407] Connection header received
0928 13:08:03.023545 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.023546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.023557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.023611 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:03.024267 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.025017 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.025892 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.026107 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.026172 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.029026 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:03.029029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:03.029029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:03.029038 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:03.030561 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:03.030927 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:03.030928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:03.030928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:03.030985 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:03.031159 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:03.031160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:03.031160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:03.031354 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:03.031435 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.031550 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.031614 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 13:08:03.033988 10280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:08:03.035643 10280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:03.043267 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.038819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:44391 (local address 127.0.0.1:50138)
0928 13:08:03.039053 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:03.039062 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:03.039119 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:03.039533 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:03.039534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:03.039537 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:03.039589 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.039592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.040984 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.040985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.042150 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.042152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.042912 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.042913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.043008 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.043024 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 13:08:03.043099 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 13:08:03.043161 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":117,"threads_started":1}
I20260928 13:08:03.043310 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.039076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.8.88.129:44391)
0928 13:08:03.039269 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.039347 (+ 78us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.039358 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:08:03.039393 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.039395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.039408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.039522 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:03.039695 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.040853 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.042243 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.042778 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.042910 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.042990 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.043128 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.043166 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260928 13:08:03.043431 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.038891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.8.88.130:44859)
0928 13:08:03.039618 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.039623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.039632 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:08:03.039671 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.039674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.039689 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.039772 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:03.039946 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.041041 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.042113 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.042886 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.042967 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.043025 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.043289 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.043331 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":76,"threads_started":1}
I20260928 13:08:03.043452 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.039076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.8.88.131:35645)
0928 13:08:03.039286 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.039292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.039302 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:08:03.039345 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.039347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.039365 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.039445 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:03.039651 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.040492 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.042181 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.042896 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.042967 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.043025 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.043310 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.043351 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
I20260928 13:08:03.043614 10781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.038790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:44859 (local address 127.0.0.1:48842)
0928 13:08:03.038995 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:03.039044 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:03.039119 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:03.039839 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:03.039840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:03.039842 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:03.039875 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.039876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.041120 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.041122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.042025 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.042027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.043065 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.043065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.043136 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.043140 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:08:03.043204 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 13:08:03.043241 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260928 13:08:03.044127 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.038819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:35645 (local address 127.0.0.1:35460)
0928 13:08:03.039226 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:03.039235 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:03.039273 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:03.039485 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:03.039489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:03.039491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:03.039586 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.039589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.040585 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.040587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.042078 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.042082 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.043065 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.043066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.043136 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.043141 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:08:03.043192 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 13:08:03.043242 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":47,"thread_start_us":112,"threads_started":1}
I20260928 13:08:03.044512 10557 tablet_service.cc:1467] Processing CreateTablet for tablet e868460272a04de495fdc62585d150bd (DEFAULT_TABLE table=test-table [id=110b6719a3654b7590cddccca700e116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:03.044485 10417 tablet_service.cc:1467] Processing CreateTablet for tablet e868460272a04de495fdc62585d150bd (DEFAULT_TABLE table=test-table [id=110b6719a3654b7590cddccca700e116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:03.044958 10417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e868460272a04de495fdc62585d150bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:03.044958 10557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e868460272a04de495fdc62585d150bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:03.044945 10697 tablet_service.cc:1467] Processing CreateTablet for tablet e868460272a04de495fdc62585d150bd (DEFAULT_TABLE table=test-table [id=110b6719a3654b7590cddccca700e116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:03.045296 10697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e868460272a04de495fdc62585d150bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:03.048012 10786 tablet_bootstrap.cc:492] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2: Bootstrap starting.
I20260928 13:08:03.048900 10788 tablet_bootstrap.cc:492] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb: Bootstrap starting.
I20260928 13:08:03.049172 10786 tablet_bootstrap.cc:654] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:03.049528 10786 log.cc:824] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:03.049575 10788 tablet_bootstrap.cc:654] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:03.049894 10788 log.cc:824] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:03.050246 10787 tablet_bootstrap.cc:492] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd: Bootstrap starting.
I20260928 13:08:03.050462 10786 tablet_bootstrap.cc:492] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2: No bootstrap required, opened a new log
I20260928 13:08:03.050640 10786 ts_tablet_manager.cc:1397] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 13:08:03.050657 10788 tablet_bootstrap.cc:492] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb: No bootstrap required, opened a new log
I20260928 13:08:03.050729 10788 ts_tablet_manager.cc:1397] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:03.050880 10787 tablet_bootstrap.cc:654] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:03.051214 10787 log.cc:824] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:03.052019 10787 tablet_bootstrap.cc:492] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd: No bootstrap required, opened a new log
I20260928 13:08:03.052094 10787 ts_tablet_manager.cc:1397] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:03.052568 10788 raft_consensus.cc:348] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } }
I20260928 13:08:03.052732 10788 raft_consensus.cc:374] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:03.052778 10788 raft_consensus.cc:729] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e40d772ccaa14d49b4dc537318a75bcb, State: Initialized, Role: FOLLOWER
I20260928 13:08:03.052796 10786 raft_consensus.cc:348] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } }
I20260928 13:08:03.052942 10788 consensus_queue.cc:260] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } }
I20260928 13:08:03.053004 10786 raft_consensus.cc:374] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:03.053042 10786 raft_consensus.cc:729] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567f476870114c22b5315ab397de9eb2, State: Initialized, Role: FOLLOWER
I20260928 13:08:03.053238 10788 ts_tablet_manager.cc:1428] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:08:03.053205 10786 consensus_queue.cc:260] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } }
I20260928 13:08:03.053406 10623 heartbeater.cc:503] Master 127.8.88.190:41029 was elected leader, sending a full tablet report...
I20260928 13:08:03.053480 10786 ts_tablet_manager.cc:1428] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:08:03.053607 10763 heartbeater.cc:503] Master 127.8.88.190:41029 was elected leader, sending a full tablet report...
I20260928 13:08:03.054258 10787 raft_consensus.cc:348] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } }
I20260928 13:08:03.054425 10787 raft_consensus.cc:374] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:03.054463 10787 raft_consensus.cc:729] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea8521539894639bb46cb94342e70fd, State: Initialized, Role: FOLLOWER
I20260928 13:08:03.054572 10787 consensus_queue.cc:260] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } }
I20260928 13:08:03.054847 10787 ts_tablet_manager.cc:1428] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:08:03.054858 10483 heartbeater.cc:503] Master 127.8.88.190:41029 was elected leader, sending a full tablet report...
W20260928 13:08:03.062984 10624 tablet.cc:2369] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:08:03.140132 10484 tablet.cc:2369] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:08:03.202607 10794 raft_consensus.cc:482] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:03.202786 10794 raft_consensus.cc:504] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } }
I20260928 13:08:03.203259 10794 leader_election.cc:288] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130:44859), 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645)
I20260928 13:08:03.206449 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.203701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:42757 (local address 127.8.88.130:44859)
0928 13:08:03.203770 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.203772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.203886 (+ 114us) server_negotiation.cc:407] Connection header received
0928 13:08:03.203971 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.203972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.203989 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.204052 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:03.204215 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.204704 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.205705 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.206063 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.206165 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.206174 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.206336 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.206376 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 13:08:03.206432 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.203706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:36509 (local address 127.8.88.131:35645)
0928 13:08:03.203805 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.203807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.203818 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:08:03.203853 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.203854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.203864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.203930 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:03.204101 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.204567 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.205675 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.206037 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.206133 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.206143 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.206313 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.206350 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 13:08:03.206497 10795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.203533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:35645 (local address 127.8.88.129:36509)
0928 13:08:03.203668 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:03.203680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:03.203771 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:03.203937 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:03.203938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:03.203940 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:03.204027 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.204029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.204662 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.204663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.205583 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.205586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.206133 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.206134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.206210 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.206215 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:08:03.206302 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 13:08:03.206348 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260928 13:08:03.206476 10796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.203533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:44859 (local address 127.8.88.129:42757)
0928 13:08:03.203787 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:03.203799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:03.203902 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:03.204057 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:03.204057 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:03.204059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:03.204139 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.204141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.204801 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.204802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.205608 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.205610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.206167 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.206168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.206249 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.206252 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:08:03.206327 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 13:08:03.206367 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":15,"thread_start_us":57,"threads_started":1}
I20260928 13:08:03.206837 10717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e868460272a04de495fdc62585d150bd" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567f476870114c22b5315ab397de9eb2" is_pre_election: true
I20260928 13:08:03.207015 10717 raft_consensus.cc:2454] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 0.
I20260928 13:08:03.207015 10577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e868460272a04de495fdc62585d150bd" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e40d772ccaa14d49b4dc537318a75bcb" is_pre_election: true
I20260928 13:08:03.207131 10577 raft_consensus.cc:2454] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 0.
I20260928 13:08:03.207273 10370 leader_election.cc:302] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 567f476870114c22b5315ab397de9eb2, dea8521539894639bb46cb94342e70fd; no voters:
I20260928 13:08:03.207417 10794 raft_consensus.cc:2781] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:03.207505 10794 raft_consensus.cc:482] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:03.207530 10794 raft_consensus.cc:3037] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:03.208287 10794 raft_consensus.cc:504] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } }
I20260928 13:08:03.208465 10794 leader_election.cc:288] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 election: Requested vote from peers e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130:44859), 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645)
I20260928 13:08:03.208658 10577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e868460272a04de495fdc62585d150bd" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e40d772ccaa14d49b4dc537318a75bcb"
I20260928 13:08:03.208645 10717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e868460272a04de495fdc62585d150bd" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567f476870114c22b5315ab397de9eb2"
I20260928 13:08:03.208745 10577 raft_consensus.cc:3037] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:03.208770 10717 raft_consensus.cc:3037] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:03.209547 10577 raft_consensus.cc:2454] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 1.
I20260928 13:08:03.209661 10717 raft_consensus.cc:2454] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 1.
I20260928 13:08:03.209787 10368 leader_election.cc:302] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dea8521539894639bb46cb94342e70fd, e40d772ccaa14d49b4dc537318a75bcb; no voters:
I20260928 13:08:03.209945 10794 raft_consensus.cc:2781] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:03.210111 10794 raft_consensus.cc:686] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [term 1 LEADER]: Becoming Leader. State: Replica: dea8521539894639bb46cb94342e70fd, State: Running, Role: LEADER
I20260928 13:08:03.210223 10794 consensus_queue.cc:237] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } }
I20260928 13:08:03.211022 10279 catalog_manager.cc:5495] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd reported cstate change: term changed from 0 to 1, leader changed from <none> to dea8521539894639bb46cb94342e70fd (127.8.88.129). New cstate: current_term: 1 leader_uuid: "dea8521539894639bb46cb94342e70fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } health_report { overall_health: HEALTHY } } }
W20260928 13:08:03.239753 10764 tablet.cc:2369] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 13:08:03 dist-test-slave-p1lc krb5kdc[10235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600882, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:03.253571 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.245786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.8.88.190:41029)
0928 13:08:03.245887 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.245890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.246015 (+ 125us) server_negotiation.cc:407] Connection header received
0928 13:08:03.246115 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.246116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.246128 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.246289 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:03.246816 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.247318 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.248045 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.248245 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.248352 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.250675 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:03.250677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:03.250678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:03.250686 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:03.252240 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:03.252680 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:03.252681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:03.252681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:03.252751 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:03.253007 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:03.253008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:03.253009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:03.253130 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:03.253209 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.253366 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.253455 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 13:08:03.253666 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.245670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:41029 (local address 127.0.0.1:46158)
0928 13:08:03.245934 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:03.245945 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:03.246037 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:03.246302 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:03.246303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:03.246636 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:03.246720 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.246723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.247407 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.247408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.247947 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.247949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.248352 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.248352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.248410 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.249027 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:03.249032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:03.250555 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:03.252361 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:03.252371 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:03.252373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:03.252579 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:03.252822 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:03.252823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:03.252823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:03.252953 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:03.253199 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:03.253208 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:03.253257 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 13:08:03.253343 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 13:08:03.253531 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":81,"thread_start_us":152,"threads_started":1}
I20260928 13:08:03.255684 10279 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:46158:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 13:08:03.255884 10279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:03.257578 10417 tablet_service.cc:1467] Processing CreateTablet for tablet 47cd8251df2444b98557e8842e6e6688 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=510857571bb54e249a6fa4772510ddf0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 13:08:03.257606 10697 tablet_service.cc:1467] Processing CreateTablet for tablet 47cd8251df2444b98557e8842e6e6688 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=510857571bb54e249a6fa4772510ddf0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 13:08:03.257616 10557 tablet_service.cc:1467] Processing CreateTablet for tablet 47cd8251df2444b98557e8842e6e6688 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=510857571bb54e249a6fa4772510ddf0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 13:08:03.257776 10697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47cd8251df2444b98557e8842e6e6688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:03.257776 10417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47cd8251df2444b98557e8842e6e6688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:03.257831 10557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47cd8251df2444b98557e8842e6e6688. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:03.259302 10788 tablet_bootstrap.cc:492] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb: Bootstrap starting.
I20260928 13:08:03.259311 10787 tablet_bootstrap.cc:492] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd: Bootstrap starting.
I20260928 13:08:03.259315 10786 tablet_bootstrap.cc:492] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2: Bootstrap starting.
I20260928 13:08:03.259773 10787 tablet_bootstrap.cc:654] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:03.259872 10786 tablet_bootstrap.cc:654] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:03.259925 10788 tablet_bootstrap.cc:654] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:03.260713 10786 tablet_bootstrap.cc:492] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2: No bootstrap required, opened a new log
I20260928 13:08:03.260782 10786 ts_tablet_manager.cc:1397] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:03.260829 10788 tablet_bootstrap.cc:492] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb: No bootstrap required, opened a new log
I20260928 13:08:03.260933 10788 ts_tablet_manager.cc:1397] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:03.261000 10786 raft_consensus.cc:348] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.261092 10786 raft_consensus.cc:374] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:03.261118 10786 raft_consensus.cc:729] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567f476870114c22b5315ab397de9eb2, State: Initialized, Role: FOLLOWER
I20260928 13:08:03.261085 10788 raft_consensus.cc:348] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.261157 10788 raft_consensus.cc:374] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:03.261183 10788 raft_consensus.cc:729] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e40d772ccaa14d49b4dc537318a75bcb, State: Initialized, Role: FOLLOWER
I20260928 13:08:03.261173 10786 consensus_queue.cc:260] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.261242 10788 consensus_queue.cc:260] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.261300 10786 ts_tablet_manager.cc:1428] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:03.261343 10788 ts_tablet_manager.cc:1428] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:03.261443 10793 tablet_replica.cc:439] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } }
I20260928 13:08:03.261431 10792 tablet_replica.cc:439] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } }
I20260928 13:08:03.261545 10787 tablet_bootstrap.cc:492] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd: No bootstrap required, opened a new log
I20260928 13:08:03.261536 10792 tablet_replica.cc:442] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:03.261536 10793 tablet_replica.cc:442] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:03.261581 10787 ts_tablet_manager.cc:1397] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:03.261744 10787 raft_consensus.cc:348] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.261828 10787 raft_consensus.cc:374] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:03.261857 10787 raft_consensus.cc:729] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea8521539894639bb46cb94342e70fd, State: Initialized, Role: FOLLOWER
I20260928 13:08:03.261905 10787 consensus_queue.cc:260] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.262006 10787 ts_tablet_manager.cc:1428] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:03.262187 10794 tablet_replica.cc:439] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } }
I20260928 13:08:03.262279 10794 tablet_replica.cc:442] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:03.306416 10794 raft_consensus.cc:482] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:03.306531 10794 raft_consensus.cc:504] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.306736 10794 leader_election.cc:288] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130:44859), 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645)
I20260928 13:08:03.306989 10577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47cd8251df2444b98557e8842e6e6688" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e40d772ccaa14d49b4dc537318a75bcb" is_pre_election: true
I20260928 13:08:03.307030 10717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47cd8251df2444b98557e8842e6e6688" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567f476870114c22b5315ab397de9eb2" is_pre_election: true
I20260928 13:08:03.307109 10577 raft_consensus.cc:2454] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 0.
I20260928 13:08:03.307149 10717 raft_consensus.cc:2454] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 0.
I20260928 13:08:03.307338 10370 leader_election.cc:302] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 567f476870114c22b5315ab397de9eb2, dea8521539894639bb46cb94342e70fd; no voters:
I20260928 13:08:03.307435 10794 raft_consensus.cc:2781] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:03.307477 10794 raft_consensus.cc:482] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:03.307521 10794 raft_consensus.cc:3037] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:03.308259 10794 raft_consensus.cc:504] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.308420 10794 leader_election.cc:288] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 election: Requested vote from peers e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130:44859), 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645)
I20260928 13:08:03.308645 10577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47cd8251df2444b98557e8842e6e6688" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e40d772ccaa14d49b4dc537318a75bcb"
I20260928 13:08:03.308738 10577 raft_consensus.cc:3037] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:03.308645 10717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47cd8251df2444b98557e8842e6e6688" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567f476870114c22b5315ab397de9eb2"
I20260928 13:08:03.308807 10717 raft_consensus.cc:3037] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:03.309685 10717 raft_consensus.cc:2454] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 1.
I20260928 13:08:03.309715 10577 raft_consensus.cc:2454] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 1.
I20260928 13:08:03.309916 10368 leader_election.cc:302] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dea8521539894639bb46cb94342e70fd, e40d772ccaa14d49b4dc537318a75bcb; no voters:
I20260928 13:08:03.310032 10794 raft_consensus.cc:2781] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:03.310088 10794 raft_consensus.cc:686] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [term 1 LEADER]: Becoming Leader. State: Replica: dea8521539894639bb46cb94342e70fd, State: Running, Role: LEADER
I20260928 13:08:03.310146 10794 consensus_queue.cc:237] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.310622 10797 tablet_replica.cc:439] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd: TxnStatusTablet state changed. Reason: New leader dea8521539894639bb46cb94342e70fd. Latest consensus state: current_term: 1 leader_uuid: "dea8521539894639bb46cb94342e70fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } }
I20260928 13:08:03.310739 10797 tablet_replica.cc:442] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd: This TxnStatusTablet replica's current role is: LEADER
I20260928 13:08:03.310951 10809 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 13:08:03.311059 10809 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 13:08:03.311048 10279 catalog_manager.cc:5495] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd reported cstate change: term changed from 0 to 1, leader changed from <none> to dea8521539894639bb46cb94342e70fd (127.8.88.129). New cstate: current_term: 1 leader_uuid: "dea8521539894639bb46cb94342e70fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } health_report { overall_health: UNKNOWN } } }
I20260928 13:08:03.325917 10804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.322462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:44391 (local address 127.0.0.1:50140)
0928 13:08:03.322587 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:03.322600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:03.322670 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:03.322880 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:03.322881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:03.322882 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 13:08:03.322943 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.322946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.324042 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.324043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.324768 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.324771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.325321 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.325322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.325381 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.325399 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 13:08:03.325732 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 13:08:03.325744 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 13:08:03.325797 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 13:08:03.325840 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 13:08:03.326002 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.322534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50140 (local address 127.8.88.129:44391)
0928 13:08:03.322602 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.322604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.322650 (+ 46us) server_negotiation.cc:407] Connection header received
0928 13:08:03.322770 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.322771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.322794 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.322866 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:03.323013 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.323944 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.324842 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.325206 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.325280 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.325503 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:03.325635 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:03.325722 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.325803 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.325828 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 13:08:03.328152 10577 raft_consensus.cc:1261] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Refusing update from remote peer dea8521539894639bb46cb94342e70fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:03.328193 10717 raft_consensus.cc:1261] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Refusing update from remote peer dea8521539894639bb46cb94342e70fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:03.328547 10794 consensus_queue.cc:1035] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:03.328692 10797 consensus_queue.cc:1035] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:03.329934 10792 tablet_replica.cc:439] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb: TxnStatusTablet state changed. Reason: New leader dea8521539894639bb46cb94342e70fd. Latest consensus state: current_term: 1 leader_uuid: "dea8521539894639bb46cb94342e70fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } }
I20260928 13:08:03.330008 10793 tablet_replica.cc:439] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2: TxnStatusTablet state changed. Reason: New leader dea8521539894639bb46cb94342e70fd. Latest consensus state: current_term: 1 leader_uuid: "dea8521539894639bb46cb94342e70fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } }
I20260928 13:08:03.330088 10792 tablet_replica.cc:442] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:03.330118 10793 tablet_replica.cc:442] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:03.331224 10797 tablet_replica.cc:439] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dea8521539894639bb46cb94342e70fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } }
I20260928 13:08:03.331334 10797 tablet_replica.cc:442] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd: This TxnStatusTablet replica's current role is: LEADER
I20260928 13:08:03.331336 10792 tablet_replica.cc:439] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dea8521539894639bb46cb94342e70fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } }
I20260928 13:08:03.331411 10792 tablet_replica.cc:442] T 47cd8251df2444b98557e8842e6e6688 P e40d772ccaa14d49b4dc537318a75bcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:03.331501 10794 tablet_replica.cc:439] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dea8521539894639bb46cb94342e70fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } }
I20260928 13:08:03.331561 10794 tablet_replica.cc:442] T 47cd8251df2444b98557e8842e6e6688 P dea8521539894639bb46cb94342e70fd: This TxnStatusTablet replica's current role is: LEADER
I20260928 13:08:03.331629 10812 mvcc.cc:205] Tried to move back new op lower bound from 7334301218109140992 to 7334301218039103488. Current Snapshot: MvccSnapshot[applied={T|T < 7334301218109140992}]
I20260928 13:08:03.331888 10813 mvcc.cc:205] Tried to move back new op lower bound from 7334301218109140992 to 7334301218039103488. Current Snapshot: MvccSnapshot[applied={T|T < 7334301218109140992 or (T in {7334301218109140992})}]
I20260928 13:08:03.332290 10793 tablet_replica.cc:439] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dea8521539894639bb46cb94342e70fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } } }
I20260928 13:08:03.332378 10793 tablet_replica.cc:442] T 47cd8251df2444b98557e8842e6e6688 P 567f476870114c22b5315ab397de9eb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:03.337749 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.334884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50148 (local address 127.8.88.129:44391)
0928 13:08:03.334929 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.334931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.334987 (+ 56us) server_negotiation.cc:407] Connection header received
0928 13:08:03.335152 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.335152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.335164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.335233 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:03.335374 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.336136 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.336923 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.337231 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.337307 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.337495 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:03.337532 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:03.337584 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.337633 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.337658 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:08:03.343684 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.340798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:44215 (local address 127.8.88.129:44391)
0928 13:08:03.340840 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.340843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.341039 (+ 196us) server_negotiation.cc:407] Connection header received
0928 13:08:03.341100 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.341100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.341109 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.341150 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:03.341288 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.341748 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.342932 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.343113 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.343176 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.343409 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:03.343446 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:03.343490 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.343573 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.343595 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:08:03.343849 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.340693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:44391 (local address 127.8.88.129:44215)
0928 13:08:03.340950 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:03.340962 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:03.341034 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:03.341163 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:03.341163 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:03.341165 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 13:08:03.341232 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.341234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.341829 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.341830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.342830 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.342834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.343193 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.343194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.343274 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.343296 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 13:08:03.343510 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 13:08:03.343511 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 13:08:03.343570 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 13:08:03.343618 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
I20260928 13:08:03.346297 10577 raft_consensus.cc:1261] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Refusing update from remote peer dea8521539894639bb46cb94342e70fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:03.346338 10717 raft_consensus.cc:1261] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Refusing update from remote peer dea8521539894639bb46cb94342e70fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:03.346509 10797 consensus_queue.cc:1035] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:03.346607 10794 consensus_queue.cc:1035] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:03.348194 10814 mvcc.cc:205] Tried to move back new op lower bound from 7334301218185052160 to 7334301217629634560. Current Snapshot: MvccSnapshot[applied={T|T < 7334301218185052160}]
I20260928 13:08:03.360133 10279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:03.360251 10279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:03.361177 10279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465: Sending DeleteTablet for 3 replicas of tablet e868460272a04de495fdc62585d150bd
I20260928 13:08:03.361532 10697 tablet_service.cc:1514] Processing DeleteTablet for tablet e868460272a04de495fdc62585d150bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:03 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:35460
I20260928 13:08:03.361644 10417 tablet_service.cc:1514] Processing DeleteTablet for tablet e868460272a04de495fdc62585d150bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:03 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:50138
I20260928 13:08:03.361800 10828 tablet_replica.cc:330] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2: stopping tablet replica
I20260928 13:08:03.361931 10828 raft_consensus.cc:2229] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:03.361996 10828 raft_consensus.cc:2258] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:03.362470 10828 ts_tablet_manager.cc:1905] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:03.362418 10557 tablet_service.cc:1514] Processing DeleteTablet for tablet e868460272a04de495fdc62585d150bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:03 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:48842
I20260928 13:08:03.362673 10830 tablet_replica.cc:330] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb: stopping tablet replica
I20260928 13:08:03.362808 10830 raft_consensus.cc:2229] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:03.362911 10830 raft_consensus.cc:2258] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:03.363051 10279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:08:03.363224 10279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:03.363731 10830 ts_tablet_manager.cc:1905] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:03.364528 10697 tablet_service.cc:1467] Processing CreateTablet for tablet 5061e231b7a64a5581a65c3d3db857c2 (DEFAULT_TABLE table=test-table [id=6ae3c1f928a646608534b4c32d989da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:03.364557 10557 tablet_service.cc:1467] Processing CreateTablet for tablet 5061e231b7a64a5581a65c3d3db857c2 (DEFAULT_TABLE table=test-table [id=6ae3c1f928a646608534b4c32d989da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:03.364570 10417 tablet_service.cc:1467] Processing CreateTablet for tablet 5061e231b7a64a5581a65c3d3db857c2 (DEFAULT_TABLE table=test-table [id=6ae3c1f928a646608534b4c32d989da5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:03.364655 10557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5061e231b7a64a5581a65c3d3db857c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:03.364692 10417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5061e231b7a64a5581a65c3d3db857c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:03.364837 10829 tablet_replica.cc:330] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd: stopping tablet replica
I20260928 13:08:03.364995 10829 raft_consensus.cc:2229] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:08:03.365091 10829 raft_consensus.cc:2258] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:03.365834 10787 tablet_bootstrap.cc:492] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd: Bootstrap starting.
I20260928 13:08:03.365859 10788 tablet_bootstrap.cc:492] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb: Bootstrap starting.
I20260928 13:08:03.366222 10697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5061e231b7a64a5581a65c3d3db857c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:03.366267 10788 tablet_bootstrap.cc:654] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:03.366266 10787 tablet_bootstrap.cc:654] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:03.367033 10787 tablet_bootstrap.cc:492] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd: No bootstrap required, opened a new log
I20260928 13:08:03.367097 10787 ts_tablet_manager.cc:1397] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:03.367125 10788 tablet_bootstrap.cc:492] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb: No bootstrap required, opened a new log
I20260928 13:08:03.367165 10788 ts_tablet_manager.cc:1397] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:03.367247 10787 raft_consensus.cc:348] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.367352 10787 raft_consensus.cc:374] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:03.367324 10788 raft_consensus.cc:348] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.367383 10787 raft_consensus.cc:729] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea8521539894639bb46cb94342e70fd, State: Initialized, Role: FOLLOWER
I20260928 13:08:03.367398 10788 raft_consensus.cc:374] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:03.367419 10788 raft_consensus.cc:729] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e40d772ccaa14d49b4dc537318a75bcb, State: Initialized, Role: FOLLOWER
I20260928 13:08:03.367426 10787 consensus_queue.cc:260] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.367466 10788 consensus_queue.cc:260] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.367525 10787 ts_tablet_manager.cc:1428] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:03.367574 10788 ts_tablet_manager.cc:1428] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 13:08:03.367642 10786 tablet_bootstrap.cc:492] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2: Bootstrap starting.
I20260928 13:08:03.367980 10829 ts_tablet_manager.cc:1905] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:03.368021 10786 tablet_bootstrap.cc:654] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:03.368528 10828 ts_tablet_manager.cc:1918] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:03.368609 10828 log.cc:1196] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/e868460272a04de495fdc62585d150bd
I20260928 13:08:03.368659 10786 tablet_bootstrap.cc:492] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2: No bootstrap required, opened a new log
I20260928 13:08:03.368719 10786 ts_tablet_manager.cc:1397] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:03.368908 10828 ts_tablet_manager.cc:1939] T e868460272a04de495fdc62585d150bd P 567f476870114c22b5315ab397de9eb2: Deleting consensus metadata
I20260928 13:08:03.368893 10786 raft_consensus.cc:348] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.368975 10786 raft_consensus.cc:374] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:03.369005 10786 raft_consensus.cc:729] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567f476870114c22b5315ab397de9eb2, State: Initialized, Role: FOLLOWER
I20260928 13:08:03.369056 10786 consensus_queue.cc:260] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.369155 10786 ts_tablet_manager.cc:1428] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:03.369431 10830 ts_tablet_manager.cc:1918] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:03.369549 10830 log.cc:1196] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/e868460272a04de495fdc62585d150bd
I20260928 13:08:03.369882 10830 ts_tablet_manager.cc:1939] T e868460272a04de495fdc62585d150bd P e40d772ccaa14d49b4dc537318a75bcb: Deleting consensus metadata
I20260928 13:08:03.370003 10829 ts_tablet_manager.cc:1918] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:03.370077 10829 log.cc:1196] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/wal/wals/e868460272a04de495fdc62585d150bd
I20260928 13:08:03.370368 10829 ts_tablet_manager.cc:1939] T e868460272a04de495fdc62585d150bd P dea8521539894639bb46cb94342e70fd: Deleting consensus metadata
I20260928 13:08:03.370494 10264 catalog_manager.cc:4841] TS e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130:44859): tablet e868460272a04de495fdc62585d150bd (table test-table [id=110b6719a3654b7590cddccca700e116]) successfully deleted
I20260928 13:08:03.370915 10267 catalog_manager.cc:4841] TS dea8521539894639bb46cb94342e70fd (127.8.88.129:44391): tablet e868460272a04de495fdc62585d150bd (table test-table [id=110b6719a3654b7590cddccca700e116]) successfully deleted
I20260928 13:08:03.377908 10266 catalog_manager.cc:4841] TS 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645): tablet e868460272a04de495fdc62585d150bd (table test-table [id=110b6719a3654b7590cddccca700e116]) successfully deleted
I20260928 13:08:03.526376 10792 raft_consensus.cc:482] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:03.526510 10792 raft_consensus.cc:504] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.526909 10792 leader_election.cc:288] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dea8521539894639bb46cb94342e70fd (127.8.88.129:44391), 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645)
I20260928 13:08:03.530648 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.527289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:36229 (local address 127.8.88.129:44391)
0928 13:08:03.527372 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.527374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.527500 (+ 126us) server_negotiation.cc:407] Connection header received
0928 13:08:03.527598 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.527599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.527609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.527683 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:03.527904 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.528628 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.529919 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.530282 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.530378 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.530392 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.530544 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.530580 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 13:08:03.530671 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.527139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:44391 (local address 127.8.88.130:36229)
0928 13:08:03.527396 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:03.527407 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:03.527520 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:03.527708 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:03.527709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:03.527711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:03.527792 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.527811 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.528728 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.528729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.529832 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.529839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.530379 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.530379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.530459 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.530463 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:08:03.530524 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 13:08:03.530568 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260928 13:08:03.531040 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.527139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:35645 (local address 127.8.88.130:50893)
0928 13:08:03.527318 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:03.527330 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:03.527429 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:03.527684 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:03.527685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:03.527687 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:03.527758 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.527760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.528438 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.528440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.529719 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.529722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.530589 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.530591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.530662 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.530667 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:08:03.530905 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 13:08:03.530949 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260928 13:08:03.531060 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.527274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:50893 (local address 127.8.88.131:35645)
0928 13:08:03.527320 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.527322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.527399 (+ 77us) server_negotiation.cc:407] Connection header received
0928 13:08:03.527542 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.527543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.527576 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.527652 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:03.527837 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.528343 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.529838 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.530459 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.530542 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.530552 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.530937 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.530989 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:08:03.531155 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5061e231b7a64a5581a65c3d3db857c2" candidate_uuid: "e40d772ccaa14d49b4dc537318a75bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea8521539894639bb46cb94342e70fd" is_pre_election: true
I20260928 13:08:03.531324 10437 raft_consensus.cc:2454] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e40d772ccaa14d49b4dc537318a75bcb in term 0.
I20260928 13:08:03.531421 10717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5061e231b7a64a5581a65c3d3db857c2" candidate_uuid: "e40d772ccaa14d49b4dc537318a75bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567f476870114c22b5315ab397de9eb2" is_pre_election: true
I20260928 13:08:03.531520 10717 raft_consensus.cc:2454] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e40d772ccaa14d49b4dc537318a75bcb in term 0.
I20260928 13:08:03.531529 10511 leader_election.cc:302] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dea8521539894639bb46cb94342e70fd, e40d772ccaa14d49b4dc537318a75bcb; no voters:
I20260928 13:08:03.531651 10792 raft_consensus.cc:2781] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:03.531675 10792 raft_consensus.cc:482] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:03.531697 10792 raft_consensus.cc:3037] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:03.532395 10792 raft_consensus.cc:504] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.532538 10792 leader_election.cc:288] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [CANDIDATE]: Term 1 election: Requested vote from peers dea8521539894639bb46cb94342e70fd (127.8.88.129:44391), 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645)
I20260928 13:08:03.532693 10717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5061e231b7a64a5581a65c3d3db857c2" candidate_uuid: "e40d772ccaa14d49b4dc537318a75bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567f476870114c22b5315ab397de9eb2"
I20260928 13:08:03.532738 10437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5061e231b7a64a5581a65c3d3db857c2" candidate_uuid: "e40d772ccaa14d49b4dc537318a75bcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea8521539894639bb46cb94342e70fd"
I20260928 13:08:03.532770 10717 raft_consensus.cc:3037] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:03.532827 10437 raft_consensus.cc:3037] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:03.533474 10717 raft_consensus.cc:2454] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e40d772ccaa14d49b4dc537318a75bcb in term 1.
I20260928 13:08:03.533656 10510 leader_election.cc:302] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 567f476870114c22b5315ab397de9eb2, e40d772ccaa14d49b4dc537318a75bcb; no voters:
I20260928 13:08:03.533685 10437 raft_consensus.cc:2454] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e40d772ccaa14d49b4dc537318a75bcb in term 1.
I20260928 13:08:03.533776 10792 raft_consensus.cc:2781] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:03.533924 10792 raft_consensus.cc:686] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [term 1 LEADER]: Becoming Leader. State: Replica: e40d772ccaa14d49b4dc537318a75bcb, State: Running, Role: LEADER
I20260928 13:08:03.534003 10792 consensus_queue.cc:237] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:03.534935 10280 catalog_manager.cc:5495] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb reported cstate change: term changed from 0 to 1, leader changed from <none> to e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130). New cstate: current_term: 1 leader_uuid: "e40d772ccaa14d49b4dc537318a75bcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } health_report { overall_health: UNKNOWN } } }
I20260928 13:08:03.555836 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.552836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48852 (local address 127.8.88.130:44859)
0928 13:08:03.552901 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.552903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.552945 (+ 42us) server_negotiation.cc:407] Connection header received
0928 13:08:03.553036 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.553037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.553054 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.553116 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:03.553255 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.553799 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.554845 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.555050 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.555106 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.555361 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:03.555533 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:03.555617 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.555681 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.555704 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:08:03.561729 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.558971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:51963 (local address 127.8.88.129:44391)
0928 13:08:03.559016 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.559019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.559082 (+ 63us) server_negotiation.cc:407] Connection header received
0928 13:08:03.559161 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.559162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.559171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.559212 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:03.559355 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.559833 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.560972 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.561176 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.561252 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.561466 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:03.561500 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:03.561543 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.561618 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.561636 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:08:03.561949 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.558865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:44391 (local address 127.8.88.130:51963)
0928 13:08:03.559023 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:03.559036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:03.559098 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:03.559223 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:03.559224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:03.559226 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 13:08:03.559300 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.559302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.559909 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.559910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.560816 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.560819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.561287 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.561288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.561355 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.561374 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 13:08:03.561562 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 13:08:03.561564 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 13:08:03.561617 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 13:08:03.561665 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260928 13:08:03.564648 10717 raft_consensus.cc:1261] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Refusing update from remote peer e40d772ccaa14d49b4dc537318a75bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:03.564831 10437 raft_consensus.cc:1261] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Refusing update from remote peer e40d772ccaa14d49b4dc537318a75bcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:03.564967 10836 consensus_queue.cc:1035] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:03.565145 10792 consensus_queue.cc:1035] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:03.574170 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.570842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:44859 (local address 127.8.88.129:48325)
0928 13:08:03.570933 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:03.570944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:03.570979 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:03.571473 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:03.571473 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:03.571475 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 13:08:03.571525 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.571527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.572234 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.572235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.573195 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:03.573200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.573729 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.573730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:03.573782 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.573788 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 13:08:03.573999 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 13:08:03.574002 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:08:03.574063 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 13:08:03.574110 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 13:08:03.574188 10783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:03.571056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:48325 (local address 127.8.88.130:44859)
0928 13:08:03.571338 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:03.571340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:03.571350 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:08:03.571374 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:03.571375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:03.571389 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:03.571447 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:03.571690 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.572158 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.573314 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:03.573602 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:03.573685 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:03.573877 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:03.573914 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:03.573968 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:03.574075 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 13:08:03.574099 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
W20260928 13:08:03.574574 10813 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 13:08:05.096666 10280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:05.096822 10280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:05.097846 10280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465: Sending DeleteTablet for 3 replicas of tablet 5061e231b7a64a5581a65c3d3db857c2
I20260928 13:08:05.098136 10697 tablet_service.cc:1514] Processing DeleteTablet for tablet 5061e231b7a64a5581a65c3d3db857c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:05 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:35460
I20260928 13:08:05.098202 10557 tablet_service.cc:1514] Processing DeleteTablet for tablet 5061e231b7a64a5581a65c3d3db857c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:05 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:48842
I20260928 13:08:05.098259 10417 tablet_service.cc:1514] Processing DeleteTablet for tablet 5061e231b7a64a5581a65c3d3db857c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:05 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:50138
I20260928 13:08:05.098501 10869 tablet_replica.cc:330] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd: stopping tablet replica
I20260928 13:08:05.098553 10869 raft_consensus.cc:2229] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:05.098591 10869 raft_consensus.cc:2258] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:05.098678 10870 tablet_replica.cc:330] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb: stopping tablet replica
I20260928 13:08:05.098757 10870 raft_consensus.cc:2229] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:08:05.098848 10870 raft_consensus.cc:2258] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:05.100910 10870 ts_tablet_manager.cc:1905] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:05.101464 10869 ts_tablet_manager.cc:1905] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:05.102775 10870 ts_tablet_manager.cc:1918] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 13:08:05.102811 10869 ts_tablet_manager.cc:1918] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 13:08:05.102847 10870 log.cc:1196] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/5061e231b7a64a5581a65c3d3db857c2
I20260928 13:08:05.102859 10869 log.cc:1196] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/wal/wals/5061e231b7a64a5581a65c3d3db857c2
I20260928 13:08:05.103075 10870 ts_tablet_manager.cc:1939] T 5061e231b7a64a5581a65c3d3db857c2 P e40d772ccaa14d49b4dc537318a75bcb: Deleting consensus metadata
I20260928 13:08:05.103075 10869 ts_tablet_manager.cc:1939] T 5061e231b7a64a5581a65c3d3db857c2 P dea8521539894639bb46cb94342e70fd: Deleting consensus metadata
I20260928 13:08:05.103332 10868 tablet_replica.cc:330] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2: stopping tablet replica
I20260928 13:08:05.103415 10868 raft_consensus.cc:2229] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:05.103462 10868 raft_consensus.cc:2258] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:05.103447 10264 catalog_manager.cc:4841] TS e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130:44859): tablet 5061e231b7a64a5581a65c3d3db857c2 (table test-table [id=6ae3c1f928a646608534b4c32d989da5]) successfully deleted
I20260928 13:08:05.103448 10267 catalog_manager.cc:4841] TS dea8521539894639bb46cb94342e70fd (127.8.88.129:44391): tablet 5061e231b7a64a5581a65c3d3db857c2 (table test-table [id=6ae3c1f928a646608534b4c32d989da5]) successfully deleted
I20260928 13:08:05.103735 10868 ts_tablet_manager.cc:1905] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:05.105369 10868 ts_tablet_manager.cc:1918] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 13:08:05.105463 10868 log.cc:1196] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/5061e231b7a64a5581a65c3d3db857c2
I20260928 13:08:05.105671 10868 ts_tablet_manager.cc:1939] T 5061e231b7a64a5581a65c3d3db857c2 P 567f476870114c22b5315ab397de9eb2: Deleting consensus metadata
I20260928 13:08:05.106066 10266 catalog_manager.cc:4841] TS 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645): tablet 5061e231b7a64a5581a65c3d3db857c2 (table test-table [id=6ae3c1f928a646608534b4c32d989da5]) successfully deleted
Sep 28 13:08:05 dist-test-slave-p1lc krb5kdc[10235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 13:08:05 dist-test-slave-p1lc krb5kdc[10235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600885, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:05.130538 10878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:05.116774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.8.88.190:41029)
0928 13:08:05.116977 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:05.116980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:05.117000 (+ 20us) server_negotiation.cc:407] Connection header received
0928 13:08:05.117030 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:05.117031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:05.117043 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:05.117128 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:05.117715 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:05.118237 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:05.119009 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:05.119231 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:05.119325 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:05.127695 (+ 8370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:05.127697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:05.127697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:05.127705 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:05.129302 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:05.129659 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:05.129661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:05.129661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:05.129723 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:05.129928 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:05.129929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:05.129929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:05.130057 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:05.130164 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:05.130315 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 13:08:05.130412 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260928 13:08:05.132051 10280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:08:05.132210 10280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:05.133599 10697 tablet_service.cc:1467] Processing CreateTablet for tablet db3647cc39b94c32952c705d811d05d6 (DEFAULT_TABLE table=test-table [id=4dc9bf00f8ae4b08a8c54e2b2cbfbe57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:05.133656 10417 tablet_service.cc:1467] Processing CreateTablet for tablet db3647cc39b94c32952c705d811d05d6 (DEFAULT_TABLE table=test-table [id=4dc9bf00f8ae4b08a8c54e2b2cbfbe57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:05.133745 10697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3647cc39b94c32952c705d811d05d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:05.133761 10417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3647cc39b94c32952c705d811d05d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:05.133702 10557 tablet_service.cc:1467] Processing CreateTablet for tablet db3647cc39b94c32952c705d811d05d6 (DEFAULT_TABLE table=test-table [id=4dc9bf00f8ae4b08a8c54e2b2cbfbe57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:05.133911 10557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3647cc39b94c32952c705d811d05d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:05.135233 10879 tablet_bootstrap.cc:492] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2: Bootstrap starting.
I20260928 13:08:05.135485 10881 tablet_bootstrap.cc:492] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb: Bootstrap starting.
I20260928 13:08:05.135679 10879 tablet_bootstrap.cc:654] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:05.135774 10880 tablet_bootstrap.cc:492] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd: Bootstrap starting.
I20260928 13:08:05.135901 10881 tablet_bootstrap.cc:654] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:05.136186 10880 tablet_bootstrap.cc:654] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:05.136397 10879 tablet_bootstrap.cc:492] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2: No bootstrap required, opened a new log
I20260928 13:08:05.136462 10879 ts_tablet_manager.cc:1397] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:05.136590 10879 raft_consensus.cc:348] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.136683 10879 raft_consensus.cc:374] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:05.136718 10879 raft_consensus.cc:729] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567f476870114c22b5315ab397de9eb2, State: Initialized, Role: FOLLOWER
I20260928 13:08:05.136781 10879 consensus_queue.cc:260] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.136837 10880 tablet_bootstrap.cc:492] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd: No bootstrap required, opened a new log
I20260928 13:08:05.136915 10880 ts_tablet_manager.cc:1397] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:05.136981 10879 ts_tablet_manager.cc:1428] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:05.137033 10880 raft_consensus.cc:348] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.137120 10880 raft_consensus.cc:374] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:05.137144 10880 raft_consensus.cc:729] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea8521539894639bb46cb94342e70fd, State: Initialized, Role: FOLLOWER
I20260928 13:08:05.137212 10880 consensus_queue.cc:260] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.137390 10880 ts_tablet_manager.cc:1428] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:05.137638 10881 tablet_bootstrap.cc:492] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb: No bootstrap required, opened a new log
I20260928 13:08:05.137703 10881 ts_tablet_manager.cc:1397] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:05.137849 10881 raft_consensus.cc:348] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.137940 10881 raft_consensus.cc:374] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:05.137974 10881 raft_consensus.cc:729] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e40d772ccaa14d49b4dc537318a75bcb, State: Initialized, Role: FOLLOWER
I20260928 13:08:05.138026 10881 consensus_queue.cc:260] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.138164 10881 ts_tablet_manager.cc:1428] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:05.308926 10797 raft_consensus.cc:482] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:05.309068 10797 raft_consensus.cc:504] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.309280 10797 leader_election.cc:288] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130:44859), 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645)
I20260928 13:08:05.309639 10577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3647cc39b94c32952c705d811d05d6" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e40d772ccaa14d49b4dc537318a75bcb" is_pre_election: true
I20260928 13:08:05.309724 10577 raft_consensus.cc:2454] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 0.
I20260928 13:08:05.309655 10716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3647cc39b94c32952c705d811d05d6" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567f476870114c22b5315ab397de9eb2" is_pre_election: true
I20260928 13:08:05.309783 10716 raft_consensus.cc:2454] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 0.
I20260928 13:08:05.309923 10368 leader_election.cc:302] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dea8521539894639bb46cb94342e70fd, e40d772ccaa14d49b4dc537318a75bcb; no voters:
I20260928 13:08:05.310063 10797 raft_consensus.cc:2781] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:05.310122 10797 raft_consensus.cc:482] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:05.310140 10797 raft_consensus.cc:3037] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:05.310866 10797 raft_consensus.cc:504] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.311029 10797 leader_election.cc:288] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 election: Requested vote from peers e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130:44859), 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645)
I20260928 13:08:05.311178 10577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3647cc39b94c32952c705d811d05d6" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e40d772ccaa14d49b4dc537318a75bcb"
I20260928 13:08:05.311180 10716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db3647cc39b94c32952c705d811d05d6" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567f476870114c22b5315ab397de9eb2"
I20260928 13:08:05.311312 10577 raft_consensus.cc:3037] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:05.311312 10716 raft_consensus.cc:3037] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:05.312120 10716 raft_consensus.cc:2454] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 1.
I20260928 13:08:05.312109 10577 raft_consensus.cc:2454] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 1.
I20260928 13:08:05.312304 10370 leader_election.cc:302] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 567f476870114c22b5315ab397de9eb2, dea8521539894639bb46cb94342e70fd; no voters:
I20260928 13:08:05.312402 10797 raft_consensus.cc:2781] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:05.312511 10797 raft_consensus.cc:686] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [term 1 LEADER]: Becoming Leader. State: Replica: dea8521539894639bb46cb94342e70fd, State: Running, Role: LEADER
I20260928 13:08:05.312574 10797 consensus_queue.cc:237] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.313341 10279 catalog_manager.cc:5495] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd reported cstate change: term changed from 0 to 1, leader changed from <none> to dea8521539894639bb46cb94342e70fd (127.8.88.129). New cstate: current_term: 1 leader_uuid: "dea8521539894639bb46cb94342e70fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } health_report { overall_health: UNKNOWN } } }
I20260928 13:08:05.325311 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:05.322654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.8.88.129:44391)
0928 13:08:05.322837 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:05.322839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:05.322846 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:08:05.322870 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:05.322876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:05.322886 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:05.322938 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:05.323100 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:05.323695 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:05.324551 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:05.324765 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:05.324835 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:05.325024 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:05.325060 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:05.325102 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:05.325152 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 13:08:05.325173 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":132,"threads_started":1}
I20260928 13:08:05.327843 10577 raft_consensus.cc:1261] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Refusing update from remote peer dea8521539894639bb46cb94342e70fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:05.327943 10716 raft_consensus.cc:1261] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Refusing update from remote peer dea8521539894639bb46cb94342e70fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:05.328119 10797 consensus_queue.cc:1035] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:05.328317 10797 consensus_queue.cc:1035] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:05.338693 10280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:05.338830 10280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:05.339527 10280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465: Sending DeleteTablet for 3 replicas of tablet db3647cc39b94c32952c705d811d05d6
I20260928 13:08:05.339797 10557 tablet_service.cc:1514] Processing DeleteTablet for tablet db3647cc39b94c32952c705d811d05d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:05 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:48842
I20260928 13:08:05.339865 10417 tablet_service.cc:1514] Processing DeleteTablet for tablet db3647cc39b94c32952c705d811d05d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:05 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:50138
I20260928 13:08:05.339957 10870 tablet_replica.cc:330] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb: stopping tablet replica
I20260928 13:08:05.340022 10870 raft_consensus.cc:2229] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:05.340068 10870 raft_consensus.cc:2258] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:05.340143 10869 tablet_replica.cc:330] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd: stopping tablet replica
I20260928 13:08:05.340282 10869 raft_consensus.cc:2229] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:08:05.340361 10869 raft_consensus.cc:2258] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:05.340409 10870 ts_tablet_manager.cc:1905] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:05.340612 10869 ts_tablet_manager.cc:1905] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:05.340708 10697 tablet_service.cc:1514] Processing DeleteTablet for tablet db3647cc39b94c32952c705d811d05d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:05 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:35460
I20260928 13:08:05.340662 10280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 13:08:05.340806 10868 tablet_replica.cc:330] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2: stopping tablet replica
W20260928 13:08:05.340806 10280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:05.340847 10868 raft_consensus.cc:2229] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:05.340909 10868 raft_consensus.cc:2258] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:05.341213 10868 ts_tablet_manager.cc:1905] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:05.342388 10557 tablet_service.cc:1467] Processing CreateTablet for tablet b222a6c09e2a4917b1122d785fde07cb (DEFAULT_TABLE table=test-table [id=c2caab1cc3204a608ad8474f8b49f8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:05.342542 10870 ts_tablet_manager.cc:1918] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:05.342597 10870 log.cc:1196] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/db3647cc39b94c32952c705d811d05d6
I20260928 13:08:05.342656 10869 ts_tablet_manager.cc:1918] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:05.342690 10697 tablet_service.cc:1467] Processing CreateTablet for tablet b222a6c09e2a4917b1122d785fde07cb (DEFAULT_TABLE table=test-table [id=c2caab1cc3204a608ad8474f8b49f8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:05.342727 10869 log.cc:1196] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-0/wal/wals/db3647cc39b94c32952c705d811d05d6
I20260928 13:08:05.342832 10870 ts_tablet_manager.cc:1939] T db3647cc39b94c32952c705d811d05d6 P e40d772ccaa14d49b4dc537318a75bcb: Deleting consensus metadata
I20260928 13:08:05.342813 10417 tablet_service.cc:1467] Processing CreateTablet for tablet b222a6c09e2a4917b1122d785fde07cb (DEFAULT_TABLE table=test-table [id=c2caab1cc3204a608ad8474f8b49f8d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:05.342913 10417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b222a6c09e2a4917b1122d785fde07cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:05.342990 10869 ts_tablet_manager.cc:1939] T db3647cc39b94c32952c705d811d05d6 P dea8521539894639bb46cb94342e70fd: Deleting consensus metadata
I20260928 13:08:05.342995 10557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b222a6c09e2a4917b1122d785fde07cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:05.343317 10267 catalog_manager.cc:4841] TS dea8521539894639bb46cb94342e70fd (127.8.88.129:44391): tablet db3647cc39b94c32952c705d811d05d6 (table test-table [id=4dc9bf00f8ae4b08a8c54e2b2cbfbe57]) successfully deleted
I20260928 13:08:05.344199 10880 tablet_bootstrap.cc:492] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd: Bootstrap starting.
I20260928 13:08:05.344199 10881 tablet_bootstrap.cc:492] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb: Bootstrap starting.
I20260928 13:08:05.344239 10868 ts_tablet_manager.cc:1918] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:05.344295 10868 log.cc:1196] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/db3647cc39b94c32952c705d811d05d6
I20260928 13:08:05.344484 10264 catalog_manager.cc:4841] TS e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130:44859): tablet db3647cc39b94c32952c705d811d05d6 (table test-table [id=4dc9bf00f8ae4b08a8c54e2b2cbfbe57]) successfully deleted
I20260928 13:08:05.344530 10868 ts_tablet_manager.cc:1939] T db3647cc39b94c32952c705d811d05d6 P 567f476870114c22b5315ab397de9eb2: Deleting consensus metadata
I20260928 13:08:05.344527 10880 tablet_bootstrap.cc:654] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:05.344697 10881 tablet_bootstrap.cc:654] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:05.344709 10697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b222a6c09e2a4917b1122d785fde07cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:05.344934 10266 catalog_manager.cc:4841] TS 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645): tablet db3647cc39b94c32952c705d811d05d6 (table test-table [id=4dc9bf00f8ae4b08a8c54e2b2cbfbe57]) successfully deleted
I20260928 13:08:05.345373 10881 tablet_bootstrap.cc:492] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb: No bootstrap required, opened a new log
I20260928 13:08:05.345449 10881 ts_tablet_manager.cc:1397] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:05.345569 10881 raft_consensus.cc:348] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.345646 10881 raft_consensus.cc:374] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:05.345678 10881 raft_consensus.cc:729] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e40d772ccaa14d49b4dc537318a75bcb, State: Initialized, Role: FOLLOWER
I20260928 13:08:05.345731 10881 consensus_queue.cc:260] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.345834 10881 ts_tablet_manager.cc:1428] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:05.345842 10879 tablet_bootstrap.cc:492] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2: Bootstrap starting.
I20260928 13:08:05.346256 10879 tablet_bootstrap.cc:654] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:05.346437 10880 tablet_bootstrap.cc:492] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd: No bootstrap required, opened a new log
I20260928 13:08:05.346506 10880 ts_tablet_manager.cc:1397] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:05.346624 10880 raft_consensus.cc:348] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.346698 10880 raft_consensus.cc:374] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:05.346731 10880 raft_consensus.cc:729] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea8521539894639bb46cb94342e70fd, State: Initialized, Role: FOLLOWER
I20260928 13:08:05.346796 10880 consensus_queue.cc:260] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.346905 10880 ts_tablet_manager.cc:1428] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:05.347464 10879 tablet_bootstrap.cc:492] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2: No bootstrap required, opened a new log
I20260928 13:08:05.347518 10879 ts_tablet_manager.cc:1397] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 13:08:05.347663 10879 raft_consensus.cc:348] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.347745 10879 raft_consensus.cc:374] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:05.347769 10879 raft_consensus.cc:729] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567f476870114c22b5315ab397de9eb2, State: Initialized, Role: FOLLOWER
I20260928 13:08:05.347816 10879 consensus_queue.cc:260] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.347911 10879 ts_tablet_manager.cc:1428] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:05.463431 10854 raft_consensus.cc:482] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:05.463550 10854 raft_consensus.cc:504] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.463739 10854 leader_election.cc:288] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130:44859), 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645)
I20260928 13:08:05.463984 10577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b222a6c09e2a4917b1122d785fde07cb" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e40d772ccaa14d49b4dc537318a75bcb" is_pre_election: true
I20260928 13:08:05.464083 10577 raft_consensus.cc:2454] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 0.
I20260928 13:08:05.464002 10716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b222a6c09e2a4917b1122d785fde07cb" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567f476870114c22b5315ab397de9eb2" is_pre_election: true
I20260928 13:08:05.464118 10716 raft_consensus.cc:2454] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 0.
I20260928 13:08:05.464267 10368 leader_election.cc:302] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dea8521539894639bb46cb94342e70fd, e40d772ccaa14d49b4dc537318a75bcb; no voters:
I20260928 13:08:05.464470 10854 raft_consensus.cc:2781] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:05.464526 10854 raft_consensus.cc:482] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:05.464562 10854 raft_consensus.cc:3037] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:05.465401 10854 raft_consensus.cc:504] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.465543 10854 leader_election.cc:288] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 election: Requested vote from peers e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130:44859), 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645)
I20260928 13:08:05.465696 10577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b222a6c09e2a4917b1122d785fde07cb" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e40d772ccaa14d49b4dc537318a75bcb"
I20260928 13:08:05.465750 10716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b222a6c09e2a4917b1122d785fde07cb" candidate_uuid: "dea8521539894639bb46cb94342e70fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567f476870114c22b5315ab397de9eb2"
I20260928 13:08:05.465799 10577 raft_consensus.cc:3037] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:05.465830 10716 raft_consensus.cc:3037] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:05.466434 10716 raft_consensus.cc:2454] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 1.
I20260928 13:08:05.466518 10577 raft_consensus.cc:2454] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dea8521539894639bb46cb94342e70fd in term 1.
I20260928 13:08:05.466616 10370 leader_election.cc:302] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 567f476870114c22b5315ab397de9eb2, dea8521539894639bb46cb94342e70fd; no voters:
I20260928 13:08:05.466749 10854 raft_consensus.cc:2781] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:05.466869 10854 raft_consensus.cc:686] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [term 1 LEADER]: Becoming Leader. State: Replica: dea8521539894639bb46cb94342e70fd, State: Running, Role: LEADER
I20260928 13:08:05.466928 10854 consensus_queue.cc:237] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } }
I20260928 13:08:05.467547 10280 catalog_manager.cc:5495] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd reported cstate change: term changed from 0 to 1, leader changed from <none> to dea8521539894639bb46cb94342e70fd (127.8.88.129). New cstate: current_term: 1 leader_uuid: "dea8521539894639bb46cb94342e70fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dea8521539894639bb46cb94342e70fd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 } health_report { overall_health: UNKNOWN } } }
I20260928 13:08:05.495645 10716 raft_consensus.cc:1261] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Refusing update from remote peer dea8521539894639bb46cb94342e70fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:05.495626 10577 raft_consensus.cc:1261] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Refusing update from remote peer dea8521539894639bb46cb94342e70fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:05.495904 10797 consensus_queue.cc:1035] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e40d772ccaa14d49b4dc537318a75bcb" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 44859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:05.496023 10849 consensus_queue.cc:1035] T b222a6c09e2a4917b1122d785fde07cb P dea8521539894639bb46cb94342e70fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "567f476870114c22b5315ab397de9eb2" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 13:08:05.500528 10847 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 13:08:07.023561 10278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:07.023736 10278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:07.024971 10278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64afc977eba94b05a2344b4de69f0465: Sending DeleteTablet for 3 replicas of tablet b222a6c09e2a4917b1122d785fde07cb
I20260928 13:08:07.025238 10416 tablet_service.cc:1514] Processing DeleteTablet for tablet b222a6c09e2a4917b1122d785fde07cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:07 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:50138
I20260928 13:08:07.025316 10557 tablet_service.cc:1514] Processing DeleteTablet for tablet b222a6c09e2a4917b1122d785fde07cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:07 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:48842
I20260928 13:08:07.025315 10697 tablet_service.cc:1514] Processing DeleteTablet for tablet b222a6c09e2a4917b1122d785fde07cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:07 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:35460
I20260928 13:08:07.025480 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 10352
I20260928 13:08:07.025554 10920 tablet_replica.cc:330] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb: stopping tablet replica
I20260928 13:08:07.026902 10920 raft_consensus.cc:2229] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:07.026978 10920 raft_consensus.cc:2258] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:07.027340 10921 tablet_replica.cc:330] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2: stopping tablet replica
I20260928 13:08:07.027432 10921 raft_consensus.cc:2229] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:07.027467 10920 ts_tablet_manager.cc:1905] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:07.027503 10921 raft_consensus.cc:2258] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:07.027809 10921 ts_tablet_manager.cc:1905] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:07.029820 10920 ts_tablet_manager.cc:1918] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 13:08:07.029937 10920 log.cc:1196] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/b222a6c09e2a4917b1122d785fde07cb
I20260928 13:08:07.029938 10921 ts_tablet_manager.cc:1918] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 13:08:07.030035 10921 log.cc:1196] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/b222a6c09e2a4917b1122d785fde07cb
I20260928 13:08:07.030194 10920 ts_tablet_manager.cc:1939] T b222a6c09e2a4917b1122d785fde07cb P e40d772ccaa14d49b4dc537318a75bcb: Deleting consensus metadata
I20260928 13:08:07.030280 10921 ts_tablet_manager.cc:1939] T b222a6c09e2a4917b1122d785fde07cb P 567f476870114c22b5315ab397de9eb2: Deleting consensus metadata
I20260928 13:08:07.030737 10266 catalog_manager.cc:4841] TS 567f476870114c22b5315ab397de9eb2 (127.8.88.131:35645): tablet b222a6c09e2a4917b1122d785fde07cb (table test-table [id=c2caab1cc3204a608ad8474f8b49f8d0]) successfully deleted
I20260928 13:08:07.030750 10264 catalog_manager.cc:4841] TS e40d772ccaa14d49b4dc537318a75bcb (127.8.88.130:44859): tablet b222a6c09e2a4917b1122d785fde07cb (table test-table [id=c2caab1cc3204a608ad8474f8b49f8d0]) successfully deleted
W20260928 13:08:07.033054 10267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:44391 (error 108)
I20260928 13:08:07.033464 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 10492
I20260928 13:08:07.037110 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:07.033556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:44391 (local address 127.0.0.1:50172)
0928 13:08:07.037017 (+ 3461us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:07.037070 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:44391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3424,"thread_start_us":69,"threads_started":1}
W20260928 13:08:07.037286 10267 catalog_manager.cc:4584] TS dea8521539894639bb46cb94342e70fd (127.8.88.129:44391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b222a6c09e2a4917b1122d785fde07cb: Network error: Client connection negotiation failed: client connection to 127.8.88.129:44391: connect: Connection refused (error 111)
I20260928 13:08:07.040135 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 10632
I20260928 13:08:07.045212 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 10248
2026-09-28T13:08:07Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5962 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:08:07 dist-test-slave-p1lc krb5kdc[10926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:08:07 dist-test-slave-p1lc krb5kdc[10926](info): set up 2 sockets
Sep 28 13:08:07 dist-test-slave-p1lc krb5kdc[10926](info): commencing operation
krb5kdc: starting...
W20260928 13:08:08.145294 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:08:08 dist-test-slave-p1lc krb5kdc[10926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:08:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:08:08.276469 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:37097
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:37097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:08.358929 10939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:08.359148 10939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:08.359171 10939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:08.360348 10939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:08.360392 10939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:08.360407 10939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:08.360420 10939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:08.360432 10939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:08.362035 10939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:37097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:08.362295 10939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:08.362547 10939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:08.367813 10945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:08.367821 10944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:08.367821 10939 server_base.cc:1034] running on GCE node
W20260928 13:08:08.368139 10947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:08.368477 10939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:08.368779 10939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:08.369927 10939 hybrid_clock.cc:648] HybridClock initialized: now 1790600888369925 us; error 42 us; skew 500 ppm
Sep 28 13:08:08 dist-test-slave-p1lc krb5kdc[10926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600888, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:08.372700 10939 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:08:08.373279 10939 webserver.cc:460] Webserver started at http://127.8.88.190:36715/ using document root <none> and password file <none>
I20260928 13:08:08.373502 10939 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:08.373546 10939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:08.373641 10939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:08.374805 10939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "b826f45a10ec40bd8493bf140f0fa77b"
format_stamp: "Formatted at 2026-09-28 13:08:08 on dist-test-slave-p1lc"
server_key: "82be2ab2c343222b664599df84a2b1fa"
server_key_iv: "1932a92e3973ef5c9c22d70ca766e2f8"
server_key_version: "encryptionkey@0"
I20260928 13:08:08.375149 10939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "b826f45a10ec40bd8493bf140f0fa77b"
format_stamp: "Formatted at 2026-09-28 13:08:08 on dist-test-slave-p1lc"
server_key: "82be2ab2c343222b664599df84a2b1fa"
server_key_iv: "1932a92e3973ef5c9c22d70ca766e2f8"
server_key_version: "encryptionkey@0"
I20260928 13:08:08.376504 10939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:08:08.377493 10954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:08.377696 10939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:08:08.377782 10939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "b826f45a10ec40bd8493bf140f0fa77b"
format_stamp: "Formatted at 2026-09-28 13:08:08 on dist-test-slave-p1lc"
server_key: "82be2ab2c343222b664599df84a2b1fa"
server_key_iv: "1932a92e3973ef5c9c22d70ca766e2f8"
server_key_version: "encryptionkey@0"
I20260928 13:08:08.377841 10939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:08.393546 10939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:08.396023 10939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:08.396143 10939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:08.400074 10939 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:37097
I20260928 13:08:08.400126 11006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:37097 every 8 connection(s)
I20260928 13:08:08.400485 10939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:08.401463 11007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:08.401520 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 10939
I20260928 13:08:08.401618 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:08.401799 8546 external_mini_cluster.cc:1428] Setting key a8940098e96908014c6fb3f5ae889bd0
I20260928 13:08:08.405083 11007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b: Bootstrap starting.
I20260928 13:08:08.405731 11007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:08.406028 11007 log.cc:824] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:08.406785 11007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b: No bootstrap required, opened a new log
Sep 28 13:08:08 dist-test-slave-p1lc krb5kdc[10926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:08.408690 11007 raft_consensus.cc:348] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b826f45a10ec40bd8493bf140f0fa77b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37097 } }
I20260928 13:08:08.408897 11007 raft_consensus.cc:374] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:08.408963 11007 raft_consensus.cc:729] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b826f45a10ec40bd8493bf140f0fa77b, State: Initialized, Role: FOLLOWER
I20260928 13:08:08.409108 11007 consensus_queue.cc:260] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b826f45a10ec40bd8493bf140f0fa77b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37097 } }
I20260928 13:08:08.409196 11007 raft_consensus.cc:388] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:08.409230 11007 raft_consensus.cc:482] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:08.409284 11007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:08.410058 11007 raft_consensus.cc:504] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b826f45a10ec40bd8493bf140f0fa77b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37097 } }
I20260928 13:08:08.410180 11007 leader_election.cc:302] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b826f45a10ec40bd8493bf140f0fa77b; no voters:
I20260928 13:08:08.410380 11007 leader_election.cc:288] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:08.410447 11012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:08.410602 11012 raft_consensus.cc:686] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [term 1 LEADER]: Becoming Leader. State: Replica: b826f45a10ec40bd8493bf140f0fa77b, State: Running, Role: LEADER
I20260928 13:08:08.410655 11007 sys_catalog.cc:564] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:08.410727 11012 consensus_queue.cc:237] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b826f45a10ec40bd8493bf140f0fa77b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37097 } }
I20260928 13:08:08.411135 11012 sys_catalog.cc:455] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b826f45a10ec40bd8493bf140f0fa77b. Latest consensus state: current_term: 1 leader_uuid: "b826f45a10ec40bd8493bf140f0fa77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b826f45a10ec40bd8493bf140f0fa77b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37097 } } }
I20260928 13:08:08.411379 11012 sys_catalog.cc:458] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:08.411298 11013 sys_catalog.cc:455] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b826f45a10ec40bd8493bf140f0fa77b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b826f45a10ec40bd8493bf140f0fa77b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37097 } } }
I20260928 13:08:08.411804 11013 sys_catalog.cc:458] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:08.411960 11021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:08.412520 11021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:08.413291 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:08.402419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.8.88.190:37097)
0928 13:08:08.402605 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:08.402616 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:08.402650 (+ 34us) server_negotiation.cc:407] Connection header received
0928 13:08:08.403158 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:08.403168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:08.403234 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:08.403431 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:08.403958 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.405312 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.406088 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.406647 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.406764 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:08.409124 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:08.409129 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:08.409129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:08.409140 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:08.411084 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:08.411570 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:08.411573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:08.411573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:08.411686 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:08.411953 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:08.411954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:08.411955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:08.412305 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:08.412448 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:08.412929 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 13:08:08.413073 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260928 13:08:08.414444 11021 catalog_manager.cc:1347] Generated new cluster ID: bc2462c4b4d84cc2adb252e690d49f4a
I20260928 13:08:08.414501 11021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:08:08.426079 11021 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:08.426828 11021 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:08.436329 11021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b826f45a10ec40bd8493bf140f0fa77b: Generated new TSK 0
I20260928 13:08:08.436573 11021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:08:08.472764 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:37097
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:08.561573 11033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:08.561843 11033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:08.561887 11033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:08.563159 11033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:08.563232 11033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:08.563292 11033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:08:08.564982 11033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:37097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:08.565299 11033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:08.565552 11033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:08.571059 11038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:08.571081 11039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:08.571283 11033 server_base.cc:1034] running on GCE node
W20260928 13:08:08.571084 11041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:08.571659 11033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:08.571907 11033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:08.573050 11033 hybrid_clock.cc:648] HybridClock initialized: now 1790600888573025 us; error 38 us; skew 500 ppm
Sep 28 13:08:08 dist-test-slave-p1lc krb5kdc[10926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600888, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:08.576083 11033 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:08:08.576571 11033 webserver.cc:460] Webserver started at http://127.8.88.129:32949/ using document root <none> and password file <none>
I20260928 13:08:08.576844 11033 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:08.576943 11033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:08.577121 11033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:08.578228 11033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "94135762bfe74e8ca6af4a95b797505f"
format_stamp: "Formatted at 2026-09-28 13:08:08 on dist-test-slave-p1lc"
server_key: "df4819f2922fdc593b0515b833c96f5c"
server_key_iv: "3cae802cc72e80c771f67a4a61ecd60e"
server_key_version: "encryptionkey@0"
I20260928 13:08:08.578604 11033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "94135762bfe74e8ca6af4a95b797505f"
format_stamp: "Formatted at 2026-09-28 13:08:08 on dist-test-slave-p1lc"
server_key: "df4819f2922fdc593b0515b833c96f5c"
server_key_iv: "3cae802cc72e80c771f67a4a61ecd60e"
server_key_version: "encryptionkey@0"
I20260928 13:08:08.579984 11033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:08:08.580857 11048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:08.581080 11033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:08.581188 11033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "94135762bfe74e8ca6af4a95b797505f"
format_stamp: "Formatted at 2026-09-28 13:08:08 on dist-test-slave-p1lc"
server_key: "df4819f2922fdc593b0515b833c96f5c"
server_key_iv: "3cae802cc72e80c771f67a4a61ecd60e"
server_key_version: "encryptionkey@0"
I20260928 13:08:08.581277 11033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:08.594172 11033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:08.596755 11033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:08.596998 11033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:08.597280 11033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:08.597637 11033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:08.597702 11033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:08.597747 11033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:08.597790 11033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:08.604549 11033 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:44469
I20260928 13:08:08.604606 11161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:44469 every 8 connection(s)
I20260928 13:08:08.605023 11033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:08:08.608409 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 11033
I20260928 13:08:08.608518 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:08:08.608690 8546 external_mini_cluster.cc:1428] Setting key f56233d8b805f673112f3f9219e34576
Sep 28 13:08:08 dist-test-slave-p1lc krb5kdc[10926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600888, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:08.617599 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:08.605971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:60553 (local address 127.8.88.190:37097)
0928 13:08:08.606081 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:08.606083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:08.606488 (+ 405us) server_negotiation.cc:407] Connection header received
0928 13:08:08.607051 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:08.607052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:08.607063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:08.607120 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:08.608312 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.609105 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.610540 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.610938 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.611025 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:08.614262 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:08.614264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:08.614264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:08.614272 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:08.615792 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:08.616359 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:08.616360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:08.616360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:08.616420 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:08.616789 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:08.616790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:08.616790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:08.616996 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:08.617066 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:08.617454 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 13:08:08.617529 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 13:08:08.617925 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:08.605901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:37097 (local address 127.8.88.129:60553)
0928 13:08:08.606227 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:08.606271 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:08.606941 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:08.607196 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:08.607200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:08.607884 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:08.608102 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:08.608107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.609247 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.609248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:08.610452 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:08.610456 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.611043 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.611044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:08.611244 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:08.611983 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:08.611996 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:08.614128 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:08.615886 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:08.615894 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:08.615897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:08.616240 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:08.616500 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:08.616501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:08.616502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:08.616663 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:08.617075 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:08.617083 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:08.617268 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 13:08:08.617416 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 13:08:08.617610 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":210,"threads_started":1}
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
I20260928 13:08:08.618692 11162 heartbeater.cc:347] Connected to a master server at 127.8.88.190:37097
I20260928 13:08:08.618782 11162 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:08.618948 11162 heartbeater.cc:511] Master 127.8.88.190:37097 requested a full tablet report, sending...
I20260928 13:08:08.619534 10971 ts_manager.cc:194] Registered new tserver with Master: 94135762bfe74e8ca6af4a95b797505f (127.8.88.129:44469)
I20260928 13:08:08.620133 10971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:60553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:08:08.638267 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:37097
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:08.722092 11167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:08.722312 11167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:08.722338 11167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:08.723568 11167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:08.723610 11167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:08.723639 11167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:08:08.725116 11167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:37097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:08.725317 11167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:08.725498 11167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:08.730681 11173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:08.730710 11172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:08.730680 11175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:08.730906 11167 server_base.cc:1034] running on GCE node
I20260928 13:08:08.731176 11167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:08.731375 11167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:08.732545 11167 hybrid_clock.cc:648] HybridClock initialized: now 1790600888732530 us; error 32 us; skew 500 ppm
Sep 28 13:08:08 dist-test-slave-p1lc krb5kdc[10926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600888, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:08.735213 11167 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:08:08.735664 11167 webserver.cc:460] Webserver started at http://127.8.88.130:34415/ using document root <none> and password file <none>
I20260928 13:08:08.735908 11167 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:08.735953 11167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:08.736106 11167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:08.737286 11167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "1a2ef944b60243faa73d901bb83c2618"
format_stamp: "Formatted at 2026-09-28 13:08:08 on dist-test-slave-p1lc"
server_key: "df1124eb0de73de1ba1d2bfc844aaa71"
server_key_iv: "8db7c8a926e768d18fc3a2240044382d"
server_key_version: "encryptionkey@0"
I20260928 13:08:08.737638 11167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "1a2ef944b60243faa73d901bb83c2618"
format_stamp: "Formatted at 2026-09-28 13:08:08 on dist-test-slave-p1lc"
server_key: "df1124eb0de73de1ba1d2bfc844aaa71"
server_key_iv: "8db7c8a926e768d18fc3a2240044382d"
server_key_version: "encryptionkey@0"
I20260928 13:08:08.738965 11167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:08:08.739816 11182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:08.740028 11167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:08.740120 11167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "1a2ef944b60243faa73d901bb83c2618"
format_stamp: "Formatted at 2026-09-28 13:08:08 on dist-test-slave-p1lc"
server_key: "df1124eb0de73de1ba1d2bfc844aaa71"
server_key_iv: "8db7c8a926e768d18fc3a2240044382d"
server_key_version: "encryptionkey@0"
I20260928 13:08:08.740198 11167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:08.767896 11167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:08.770478 11167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:08.770597 11167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:08.770778 11167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:08.771164 11167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:08.771206 11167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:08.771255 11167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:08.771283 11167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:08.777441 11167 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:42739
I20260928 13:08:08.777486 11295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:42739 every 8 connection(s)
I20260928 13:08:08.777812 11167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:08:08.783979 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 11167
I20260928 13:08:08.784116 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:08:08.784376 8546 external_mini_cluster.cc:1428] Setting key f53b0ec127cd17cb903701d6ae60805b
Sep 28 13:08:08 dist-test-slave-p1lc krb5kdc[10926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600888, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:08.788473 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:08.778594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:56551 (local address 127.8.88.190:37097)
0928 13:08:08.778660 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:08.778663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:08.778806 (+ 143us) server_negotiation.cc:407] Connection header received
0928 13:08:08.779301 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:08.779302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:08.779311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:08.779376 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:08.780495 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.781114 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.782201 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.782386 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.782440 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:08.785197 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:08.785199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:08.785199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:08.785207 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:08.786980 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:08.787415 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:08.787416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:08.787417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:08.787491 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:08.787774 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:08.787775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:08.787775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:08.787897 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:08.787971 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:08.788287 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 13:08:08.788373 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 13:08:08.788698 11298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:08.778513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:37097 (local address 127.8.88.130:56551)
0928 13:08:08.778670 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:08.778714 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:08.779215 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:08.779520 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:08.779523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:08.780171 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:08.780407 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:08.780413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.781210 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.781211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:08.782125 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:08.782127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.782483 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.782484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:08.782598 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:08.783150 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:08.783158 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:08.785089 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:08.787073 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:08.787082 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:08.787085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:08.787334 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:08.787570 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:08.787571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:08.787571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:08.787690 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:08.787973 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:08.787980 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:08.788128 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 13:08:08.788267 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 13:08:08.788441 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260928 13:08:08.789407 11296 heartbeater.cc:347] Connected to a master server at 127.8.88.190:37097
I20260928 13:08:08.789511 11296 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:08.789691 11296 heartbeater.cc:511] Master 127.8.88.190:37097 requested a full tablet report, sending...
I20260928 13:08:08.790187 10971 ts_manager.cc:194] Registered new tserver with Master: 1a2ef944b60243faa73d901bb83c2618 (127.8.88.130:42739)
I20260928 13:08:08.790637 10971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:56551
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:08:08.812697 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:37097
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:38407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:08.899101 11301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:08.899313 11301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:08.899338 11301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:08.900498 11301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:08.900537 11301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:08.900563 11301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:08:08.901968 11301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:38407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:37097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:08.902168 11301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:08.902344 11301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:08.907473 11306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:08.907423 11309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:08.907476 11307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:08.907806 11301 server_base.cc:1034] running on GCE node
I20260928 13:08:08.908032 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:08.908262 11301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:08.909457 11301 hybrid_clock.cc:648] HybridClock initialized: now 1790600888909448 us; error 42 us; skew 500 ppm
Sep 28 13:08:08 dist-test-slave-p1lc krb5kdc[10926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600888, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:08.912550 11301 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:08:08.913139 11301 webserver.cc:460] Webserver started at http://127.8.88.131:39283/ using document root <none> and password file <none>
I20260928 13:08:08.913391 11301 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:08.913466 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:08.913616 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:08.914721 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "71e7c87cee3046f3940e3b861ae06331"
format_stamp: "Formatted at 2026-09-28 13:08:08 on dist-test-slave-p1lc"
server_key: "dfaf401688f493785e66cd15bebce86c"
server_key_iv: "6177113ef9d588deffade83e170740ce"
server_key_version: "encryptionkey@0"
I20260928 13:08:08.915104 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "71e7c87cee3046f3940e3b861ae06331"
format_stamp: "Formatted at 2026-09-28 13:08:08 on dist-test-slave-p1lc"
server_key: "dfaf401688f493785e66cd15bebce86c"
server_key_iv: "6177113ef9d588deffade83e170740ce"
server_key_version: "encryptionkey@0"
I20260928 13:08:08.916567 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:08:08.917526 11316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:08.917730 11301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:08:08.917836 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "71e7c87cee3046f3940e3b861ae06331"
format_stamp: "Formatted at 2026-09-28 13:08:08 on dist-test-slave-p1lc"
server_key: "dfaf401688f493785e66cd15bebce86c"
server_key_iv: "6177113ef9d588deffade83e170740ce"
server_key_version: "encryptionkey@0"
I20260928 13:08:08.917941 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:08.938742 11301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:08.941248 11301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:08.941430 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:08.941676 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:08.942039 11301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:08.942106 11301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:08.942147 11301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:08.942194 11301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:08.948643 11301 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:45353
I20260928 13:08:08.948724 11429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:45353 every 8 connection(s)
I20260928 13:08:08.949112 11301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
Sep 28 13:08:08 dist-test-slave-p1lc krb5kdc[10926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600888, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:08.958220 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 11301
I20260928 13:08:08.958333 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:08:08.958614 8546 external_mini_cluster.cc:1428] Setting key f5856a3ca2deb952744ce73f9496c246
I20260928 13:08:08.959203 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:08.950044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:38597 (local address 127.8.88.190:37097)
0928 13:08:08.950114 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:08.950116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:08.950305 (+ 189us) server_negotiation.cc:407] Connection header received
0928 13:08:08.951327 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:08.951328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:08.951339 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:08.951407 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:08.952038 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.952549 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.953624 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.953806 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.953861 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:08.956144 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:08.956145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:08.956146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:08.956153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:08.957503 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:08.957914 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:08.957915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:08.957915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:08.957986 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:08.958370 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:08.958371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:08.958372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:08.958548 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:08.958618 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:08.959017 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 13:08:08.959110 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 13:08:08.959431 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:08.950032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:37097 (local address 127.8.88.131:38597)
0928 13:08:08.950195 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:08.950234 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:08.951220 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:08.951467 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:08.951471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:08.951736 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:08.951941 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:08.951945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.952660 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.952661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:08.953549 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:08.953551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:08.953905 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:08.953906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:08.954021 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:08.954636 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:08.954645 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:08.956052 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:08.957582 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:08.957589 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:08.957590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:08.957814 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:08.958092 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:08.958094 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:08.958094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:08.958261 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:08.958630 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:08.958638 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:08.958839 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 13:08:08.958976 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 13:08:08.959159 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260928 13:08:08.959923 11430 heartbeater.cc:347] Connected to a master server at 127.8.88.190:37097
I20260928 13:08:08.960032 11430 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:08.960182 11430 heartbeater.cc:511] Master 127.8.88.190:37097 requested a full tablet report, sending...
I20260928 13:08:08.960587 10971 ts_manager.cc:194] Registered new tserver with Master: 71e7c87cee3046f3940e3b861ae06331 (127.8.88.131:45353)
I20260928 13:08:08.961009 10971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:38597
I20260928 13:08:08.971297 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 13:08:08.983358 11010 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:08:08.982164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50066 (local address 127.8.88.190:37097)
0928 13:08:08.982275 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:08.982278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:08.982394 (+ 116us) server_negotiation.cc:407] Connection header received
0928 13:08:08.982529 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:08.982530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:08.982548 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:08.982612 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:08.983258 (+ 646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50066: BlockingRecv error: recv got EOF from 127.0.0.1:50066 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 13:08:08.984262 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 11033
I20260928 13:08:08.989063 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 11167
I20260928 13:08:08.993893 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 11301
I20260928 13:08:08.998802 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 10939
2026-09-28T13:08:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1952 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:08:09 dist-test-slave-p1lc krb5kdc[11442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:08:09 dist-test-slave-p1lc krb5kdc[11442](info): set up 2 sockets
Sep 28 13:08:09 dist-test-slave-p1lc krb5kdc[11442](info): commencing operation
krb5kdc: starting...
W20260928 13:08:10.061204 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:08:10 dist-test-slave-p1lc krb5kdc[11442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:08:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:08:10.188844 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:38871
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:38871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:10.272797 11455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:10.273062 11455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:10.273087 11455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:10.274310 11455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:10.274364 11455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:10.274379 11455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:10.274391 11455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:10.274405 11455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:10.275982 11455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:38871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:38871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:10.276271 11455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:10.276572 11455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:10.281813 11460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:10.281875 11461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:10.281975 11463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:10.282003 11455 server_base.cc:1034] running on GCE node
I20260928 13:08:10.282394 11455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:10.282698 11455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:10.283871 11455 hybrid_clock.cc:648] HybridClock initialized: now 1790600890283861 us; error 42 us; skew 500 ppm
Sep 28 13:08:10 dist-test-slave-p1lc krb5kdc[11442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600890, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:10.292603 11455 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:08:10.293182 11455 webserver.cc:460] Webserver started at http://127.8.88.190:35975/ using document root <none> and password file <none>
I20260928 13:08:10.293438 11455 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:10.293511 11455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:10.293663 11455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:10.294793 11455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "f534f2896a494fb5afdc19b8970b03a2"
format_stamp: "Formatted at 2026-09-28 13:08:10 on dist-test-slave-p1lc"
server_key: "7b7606b97144f5dba1e5c1c9ab07f509"
server_key_iv: "568f5e65d66afe386bec25f622d335ee"
server_key_version: "encryptionkey@0"
I20260928 13:08:10.295173 11455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "f534f2896a494fb5afdc19b8970b03a2"
format_stamp: "Formatted at 2026-09-28 13:08:10 on dist-test-slave-p1lc"
server_key: "7b7606b97144f5dba1e5c1c9ab07f509"
server_key_iv: "568f5e65d66afe386bec25f622d335ee"
server_key_version: "encryptionkey@0"
I20260928 13:08:10.296558 11455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:08:10.297565 11470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:10.297803 11455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:08:10.297915 11455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "f534f2896a494fb5afdc19b8970b03a2"
format_stamp: "Formatted at 2026-09-28 13:08:10 on dist-test-slave-p1lc"
server_key: "7b7606b97144f5dba1e5c1c9ab07f509"
server_key_iv: "568f5e65d66afe386bec25f622d335ee"
server_key_version: "encryptionkey@0"
I20260928 13:08:10.298020 11455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:10.317574 11455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:10.320184 11455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:10.320465 11455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:10.325273 11455 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:38871
I20260928 13:08:10.325314 11522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:38871 every 8 connection(s)
I20260928 13:08:10.325704 11455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:10.326622 11523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:10.329310 11523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2: Bootstrap starting.
I20260928 13:08:10.329938 11523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:10.330245 11523 log.cc:824] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:10.331002 11523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2: No bootstrap required, opened a new log
I20260928 13:08:10.332669 11523 raft_consensus.cc:348] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f534f2896a494fb5afdc19b8970b03a2" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38871 } }
I20260928 13:08:10.332813 11523 raft_consensus.cc:374] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:10.332835 11523 raft_consensus.cc:729] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f534f2896a494fb5afdc19b8970b03a2, State: Initialized, Role: FOLLOWER
I20260928 13:08:10.333026 11523 consensus_queue.cc:260] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f534f2896a494fb5afdc19b8970b03a2" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38871 } }
I20260928 13:08:10.333104 11523 raft_consensus.cc:388] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:10.333128 11523 raft_consensus.cc:482] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:10.333184 11523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:10.333999 11523 raft_consensus.cc:504] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f534f2896a494fb5afdc19b8970b03a2" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38871 } }
I20260928 13:08:10.334090 11523 leader_election.cc:302] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f534f2896a494fb5afdc19b8970b03a2; no voters:
I20260928 13:08:10.334268 11523 leader_election.cc:288] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:10.334401 11526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:10.334553 11526 raft_consensus.cc:686] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [term 1 LEADER]: Becoming Leader. State: Replica: f534f2896a494fb5afdc19b8970b03a2, State: Running, Role: LEADER
I20260928 13:08:10.334568 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 11455
I20260928 13:08:10.334559 11523 sys_catalog.cc:564] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:10.334656 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:10.334666 11526 consensus_queue.cc:237] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f534f2896a494fb5afdc19b8970b03a2" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38871 } }
I20260928 13:08:10.334867 8546 external_mini_cluster.cc:1428] Setting key 515c2c935b6edff18bcfebe3812ddf23
I20260928 13:08:10.335104 11526 sys_catalog.cc:455] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f534f2896a494fb5afdc19b8970b03a2. Latest consensus state: current_term: 1 leader_uuid: "f534f2896a494fb5afdc19b8970b03a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f534f2896a494fb5afdc19b8970b03a2" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38871 } } }
I20260928 13:08:10.335117 11527 sys_catalog.cc:455] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f534f2896a494fb5afdc19b8970b03a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f534f2896a494fb5afdc19b8970b03a2" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38871 } } }
I20260928 13:08:10.335197 11526 sys_catalog.cc:458] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:10.335227 11527 sys_catalog.cc:458] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:10.335976 11535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:10.336624 11535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:10.338325 11535 catalog_manager.cc:1347] Generated new cluster ID: 1b66aa0216b0432fb77f25b0a261e6ff
I20260928 13:08:10.338385 11535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:08:10 dist-test-slave-p1lc krb5kdc[11442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:10.347087 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:10.335469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43206 (local address 127.8.88.190:38871)
0928 13:08:10.335636 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:10.335646 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:10.335834 (+ 188us) server_negotiation.cc:407] Connection header received
0928 13:08:10.335963 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:10.335970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:10.336036 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:10.336181 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:10.337195 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:10.338423 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:10.339479 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:10.340125 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:10.340260 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:10.343136 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:10.343141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:10.343142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:10.343155 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:10.345205 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:10.345638 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:10.345641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:10.345641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:10.345758 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:10.346013 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:10.346014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:10.346014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:10.346396 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:10.346555 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:10.346678 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 13:08:10.346822 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260928 13:08:10.348083 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:08:10.348860 11535 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:10.349608 11535 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:10.353508 11535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f534f2896a494fb5afdc19b8970b03a2: Generated new TSK 0
I20260928 13:08:10.353703 11535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:08:10.364593 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:10.360634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.8.88.190:38871)
0928 13:08:10.360701 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:10.360704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:10.360807 (+ 103us) server_negotiation.cc:407] Connection header received
0928 13:08:10.360931 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:10.360932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:10.360961 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:10.361012 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:10.361462 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:10.361998 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:10.363101 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:10.363323 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:10.363382 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:10.364096 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:10.364097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:10.364098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:10.364288 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:10.364305 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:10.364311 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:10.364357 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:10.364451 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 13:08:10.364476 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 13:08:10.367170 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 11455
2026-09-28T13:08:10Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1366 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:08:10 dist-test-slave-p1lc krb5kdc[11556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:08:10 dist-test-slave-p1lc krb5kdc[11556](info): set up 2 sockets
Sep 28 13:08:10 dist-test-slave-p1lc krb5kdc[11556](info): commencing operation
krb5kdc: starting...
W20260928 13:08:11.425086 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:08:11 dist-test-slave-p1lc krb5kdc[11556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:08:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:08:11.553089 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:36699
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:36699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:11.638446 11569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:11.638677 11569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:11.638700 11569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:11.639918 11569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:11.639967 11569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:11.639982 11569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:11.639995 11569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:11.640049 11569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:11.641638 11569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:36699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:36699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:11.641929 11569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:11.642177 11569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:11.647444 11577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:11.647452 11575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:11.647475 11574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:11.647476 11569 server_base.cc:1034] running on GCE node
I20260928 13:08:11.647994 11569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:11.648322 11569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:11.649508 11569 hybrid_clock.cc:648] HybridClock initialized: now 1790600891649455 us; error 38 us; skew 500 ppm
Sep 28 13:08:11 dist-test-slave-p1lc krb5kdc[11556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600891, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:11.652284 11569 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:08:11.652717 11569 webserver.cc:460] Webserver started at http://127.8.88.190:37089/ using document root <none> and password file <none>
I20260928 13:08:11.652987 11569 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:11.653038 11569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:11.653133 11569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:11.654301 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "6772b807cdb348bcac52638259b3998a"
format_stamp: "Formatted at 2026-09-28 13:08:11 on dist-test-slave-p1lc"
server_key: "5006299d996a5beaa0da18e5eff01d50"
server_key_iv: "27a14f6e14cc343a765d50a82517cbfa"
server_key_version: "encryptionkey@0"
I20260928 13:08:11.654626 11569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "6772b807cdb348bcac52638259b3998a"
format_stamp: "Formatted at 2026-09-28 13:08:11 on dist-test-slave-p1lc"
server_key: "5006299d996a5beaa0da18e5eff01d50"
server_key_iv: "27a14f6e14cc343a765d50a82517cbfa"
server_key_version: "encryptionkey@0"
I20260928 13:08:11.655931 11569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:08:11.656767 11584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:11.656982 11569 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:11.657104 11569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "6772b807cdb348bcac52638259b3998a"
format_stamp: "Formatted at 2026-09-28 13:08:11 on dist-test-slave-p1lc"
server_key: "5006299d996a5beaa0da18e5eff01d50"
server_key_iv: "27a14f6e14cc343a765d50a82517cbfa"
server_key_version: "encryptionkey@0"
I20260928 13:08:11.657183 11569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:11.664525 11569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:11.666635 11569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:11.666805 11569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:11.671010 11569 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:36699
I20260928 13:08:11.671038 11636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:36699 every 8 connection(s)
I20260928 13:08:11.671404 11569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:11.672555 11637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:11.674925 11637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a: Bootstrap starting.
I20260928 13:08:11.675477 11637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:11.675706 11637 log.cc:824] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:11.676393 11637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a: No bootstrap required, opened a new log
I20260928 13:08:11.678054 11637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6772b807cdb348bcac52638259b3998a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36699 } }
I20260928 13:08:11.678193 11637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:11.678216 11637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6772b807cdb348bcac52638259b3998a, State: Initialized, Role: FOLLOWER
I20260928 13:08:11.678362 11637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6772b807cdb348bcac52638259b3998a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36699 } }
I20260928 13:08:11.678443 11637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:11.678465 11637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:11.678493 11637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:11.678700 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 11569
I20260928 13:08:11.678807 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:11.678987 8546 external_mini_cluster.cc:1428] Setting key 7a2c03b7b34071c08af032cfc5da377a
I20260928 13:08:11.679649 11637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6772b807cdb348bcac52638259b3998a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36699 } }
I20260928 13:08:11.679769 11637 leader_election.cc:302] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6772b807cdb348bcac52638259b3998a; no voters:
I20260928 13:08:11.679932 11637 leader_election.cc:288] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:11.679952 11642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:11.680063 11642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [term 1 LEADER]: Becoming Leader. State: Replica: 6772b807cdb348bcac52638259b3998a, State: Running, Role: LEADER
I20260928 13:08:11.680147 11642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6772b807cdb348bcac52638259b3998a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36699 } }
I20260928 13:08:11.680195 11637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:11.680454 11644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6772b807cdb348bcac52638259b3998a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6772b807cdb348bcac52638259b3998a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36699 } } }
I20260928 13:08:11.680537 11644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:11.680454 11643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6772b807cdb348bcac52638259b3998a. Latest consensus state: current_term: 1 leader_uuid: "6772b807cdb348bcac52638259b3998a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6772b807cdb348bcac52638259b3998a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36699 } } }
I20260928 13:08:11.680711 11643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:11.680753 11646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:11.681342 11646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:11.683017 11646 catalog_manager.cc:1347] Generated new cluster ID: bbb3ac4ccea545edae27ef42bd61eea8
I20260928 13:08:11.683071 11646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:08:11 dist-test-slave-p1lc krb5kdc[11556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:11.691960 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:11.679568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40584 (local address 127.8.88.190:36699)
0928 13:08:11.679736 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:11.679747 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:11.679782 (+ 35us) server_negotiation.cc:407] Connection header received
0928 13:08:11.679942 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:11.679948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:11.680056 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:11.681043 (+ 987us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:11.681176 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:11.682739 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:11.683820 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:11.684454 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:11.684572 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:11.687410 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:11.687414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:11.687415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:11.687427 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:11.690076 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:11.690499 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:11.690503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:11.690503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:11.690621 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:11.690846 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:11.690846 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:11.690847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:11.691167 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:11.691309 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:11.691444 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 13:08:11.691581 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260928 13:08:11.700038 11646 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:11.700651 11646 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:11.709249 11646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6772b807cdb348bcac52638259b3998a: Generated new TSK 0
I20260928 13:08:11.709525 11646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:08:11.740108 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36699
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:11.840384 11663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:11.840605 11663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:11.840631 11663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:11.841825 11663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:11.841866 11663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:11.841894 11663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:08:11.843318 11663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:11.843582 11663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:11.843822 11663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:11.849175 11668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:11.849290 11663 server_base.cc:1034] running on GCE node
W20260928 13:08:11.849237 11669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:11.849175 11671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:11.849620 11663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:11.849881 11663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:11.851058 11663 hybrid_clock.cc:648] HybridClock initialized: now 1790600891851035 us; error 42 us; skew 500 ppm
Sep 28 13:08:11 dist-test-slave-p1lc krb5kdc[11556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600891, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:11.854025 11663 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:08:11.854511 11663 webserver.cc:460] Webserver started at http://127.8.88.129:42655/ using document root <none> and password file <none>
I20260928 13:08:11.854746 11663 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:11.854794 11663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:11.854997 11663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:11.856132 11663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "122ca125d7d042d9b967cfd0fe133760"
format_stamp: "Formatted at 2026-09-28 13:08:11 on dist-test-slave-p1lc"
server_key: "ed7db39b0629f816935dce4ffb775a41"
server_key_iv: "71fce91aa157660dc38c66497fd930b7"
server_key_version: "encryptionkey@0"
I20260928 13:08:11.856503 11663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "122ca125d7d042d9b967cfd0fe133760"
format_stamp: "Formatted at 2026-09-28 13:08:11 on dist-test-slave-p1lc"
server_key: "ed7db39b0629f816935dce4ffb775a41"
server_key_iv: "71fce91aa157660dc38c66497fd930b7"
server_key_version: "encryptionkey@0"
I20260928 13:08:11.858263 11663 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 13:08:11.859220 11678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:11.859555 11663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:11.859687 11663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "122ca125d7d042d9b967cfd0fe133760"
format_stamp: "Formatted at 2026-09-28 13:08:11 on dist-test-slave-p1lc"
server_key: "ed7db39b0629f816935dce4ffb775a41"
server_key_iv: "71fce91aa157660dc38c66497fd930b7"
server_key_version: "encryptionkey@0"
I20260928 13:08:11.859805 11663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:11.883450 11663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:11.885866 11663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:11.886077 11663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:11.886343 11663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:11.886669 11663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:11.886727 11663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:11.886771 11663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:11.886801 11663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:11.893069 11663 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:45227
I20260928 13:08:11.893113 11791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:45227 every 8 connection(s)
I20260928 13:08:11.893440 11663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:08:11.896370 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 11663
I20260928 13:08:11.896512 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:08:11.896687 8546 external_mini_cluster.cc:1428] Setting key c75799b12c03d23cb977e465d15d706b
Sep 28 13:08:11 dist-test-slave-p1lc krb5kdc[11556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600891, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:11.904377 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:11.894236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:42279 (local address 127.8.88.190:36699)
0928 13:08:11.894305 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:11.894307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:11.894573 (+ 266us) server_negotiation.cc:407] Connection header received
0928 13:08:11.895062 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:11.895063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:11.895072 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:11.895134 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:11.896171 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:11.896733 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:11.898229 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:11.898425 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:11.898489 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:11.900924 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:11.900926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:11.900927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:11.900934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:11.902599 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:11.903268 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:11.903269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:11.903270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:11.903336 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:11.903645 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:11.903646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:11.903646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:11.903774 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:11.903854 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:11.904171 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 13:08:11.904279 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 13:08:11.904594 11794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:11.894228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:36699 (local address 127.8.88.129:42279)
0928 13:08:11.894399 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:11.894442 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:11.894971 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:11.895230 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:11.895235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:11.895868 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:11.896076 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:11.896082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:11.896841 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:11.896843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:11.898136 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:11.898140 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:11.898513 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:11.898514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:11.898671 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:11.899332 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:11.899341 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:11.900783 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:11.902877 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:11.902911 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:11.902914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:11.903157 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:11.903430 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:11.903431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:11.903432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:11.903563 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:11.903843 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:11.903850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:11.904016 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 13:08:11.904147 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 13:08:11.904334 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
I20260928 13:08:11.905202 11792 heartbeater.cc:347] Connected to a master server at 127.8.88.190:36699
I20260928 13:08:11.905364 11792 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:11.905579 11792 heartbeater.cc:511] Master 127.8.88.190:36699 requested a full tablet report, sending...
I20260928 13:08:11.906176 11601 ts_manager.cc:194] Registered new tserver with Master: 122ca125d7d042d9b967cfd0fe133760 (127.8.88.129:45227)
I20260928 13:08:11.906896 11601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:42279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:08:11.925549 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36699
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:12.014844 11797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:12.015106 11797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:12.015136 11797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:12.016527 11797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:12.016587 11797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:12.016616 11797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:08:12.018319 11797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:12.018608 11797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:12.018905 11797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:12.024318 11803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:12.024322 11805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:12.024446 11797 server_base.cc:1034] running on GCE node
W20260928 13:08:12.024320 11802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:12.024784 11797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:12.025087 11797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:12.026336 11797 hybrid_clock.cc:648] HybridClock initialized: now 1790600892026320 us; error 54 us; skew 500 ppm
Sep 28 13:08:12 dist-test-slave-p1lc krb5kdc[11556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:12.029145 11797 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:08:12.029592 11797 webserver.cc:460] Webserver started at http://127.8.88.130:43067/ using document root <none> and password file <none>
I20260928 13:08:12.029834 11797 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:12.029879 11797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:12.029973 11797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:12.031033 11797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "3fa1d9a5328446e082cca1d8bb6d6204"
format_stamp: "Formatted at 2026-09-28 13:08:12 on dist-test-slave-p1lc"
server_key: "8302aac9f0f88578a0aa24edc08b7ca4"
server_key_iv: "ee5a9cdf79be591a10c2dfc540896a45"
server_key_version: "encryptionkey@0"
I20260928 13:08:12.031361 11797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "3fa1d9a5328446e082cca1d8bb6d6204"
format_stamp: "Formatted at 2026-09-28 13:08:12 on dist-test-slave-p1lc"
server_key: "8302aac9f0f88578a0aa24edc08b7ca4"
server_key_iv: "ee5a9cdf79be591a10c2dfc540896a45"
server_key_version: "encryptionkey@0"
I20260928 13:08:12.033047 11797 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 13:08:12.034046 11812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:12.034276 11797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:12.034360 11797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "3fa1d9a5328446e082cca1d8bb6d6204"
format_stamp: "Formatted at 2026-09-28 13:08:12 on dist-test-slave-p1lc"
server_key: "8302aac9f0f88578a0aa24edc08b7ca4"
server_key_iv: "ee5a9cdf79be591a10c2dfc540896a45"
server_key_version: "encryptionkey@0"
I20260928 13:08:12.034421 11797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:12.045779 11797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:12.048249 11797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:12.048379 11797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:12.048585 11797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:12.048975 11797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:12.049017 11797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:12.049042 11797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:12.049072 11797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:12.055315 11797 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:46487
I20260928 13:08:12.055459 11925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:46487 every 8 connection(s)
I20260928 13:08:12.055780 11797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:08:12.061434 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 11797
I20260928 13:08:12.061527 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:08:12.061669 8546 external_mini_cluster.cc:1428] Setting key a92880e3dad2af528a800ec7eaa1568e
Sep 28 13:08:12 dist-test-slave-p1lc krb5kdc[11556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:12.067848 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:12.056704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:60679 (local address 127.8.88.190:36699)
0928 13:08:12.056789 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:12.056791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:12.057016 (+ 225us) server_negotiation.cc:407] Connection header received
0928 13:08:12.057759 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:12.057761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:12.057773 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:12.057879 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:12.059254 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:12.059903 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:12.061096 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:12.061294 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:12.061481 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:12.064524 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:12.064526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:12.064526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:12.064536 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:12.066091 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:12.066694 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:12.066695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:12.066695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:12.066748 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:12.067079 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:12.067080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:12.067081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:12.067218 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:12.067284 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:12.067685 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 13:08:12.067778 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:08:12.068162 11928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:12.056696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:36699 (local address 127.8.88.130:60679)
0928 13:08:12.056887 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:12.056929 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:12.057636 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:12.058025 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:12.058029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:12.058904 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:12.059116 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:12.059122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:12.060024 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:12.060024 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:12.060995 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:12.060998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:12.061492 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:12.061494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:12.061657 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:12.062622 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:12.062634 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:12.064422 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:12.066187 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:12.066195 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:12.066198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:12.066614 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:12.066823 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:12.066825 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:12.066825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:12.066986 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:12.067300 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:12.067308 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:12.067504 (+ 196us) client_negotiation.cc:769] Sending connection context
0928 13:08:12.067641 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 13:08:12.067847 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260928 13:08:12.068717 11926 heartbeater.cc:347] Connected to a master server at 127.8.88.190:36699
I20260928 13:08:12.068847 11926 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:12.069070 11926 heartbeater.cc:511] Master 127.8.88.190:36699 requested a full tablet report, sending...
I20260928 13:08:12.069490 11601 ts_manager.cc:194] Registered new tserver with Master: 3fa1d9a5328446e082cca1d8bb6d6204 (127.8.88.130:46487)
I20260928 13:08:12.069841 11601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:60679
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:08:12.091986 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36699
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:12.174081 11931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:12.174366 11931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:12.174409 11931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:12.175617 11931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:12.175688 11931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:12.175734 11931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:08:12.177330 11931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:12.177619 11931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:12.177860 11931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:12.183116 11936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:12.183120 11937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:12.183151 11939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:12.183461 11931 server_base.cc:1034] running on GCE node
I20260928 13:08:12.183667 11931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:12.183920 11931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:12.185108 11931 hybrid_clock.cc:648] HybridClock initialized: now 1790600892185086 us; error 37 us; skew 500 ppm
Sep 28 13:08:12 dist-test-slave-p1lc krb5kdc[11556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:12.187673 11931 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:08:12.188140 11931 webserver.cc:460] Webserver started at http://127.8.88.131:34683/ using document root <none> and password file <none>
I20260928 13:08:12.188400 11931 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:12.188475 11931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:12.188621 11931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:12.189769 11931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "88ed2c48bccf409a8440ec3b495a2642"
format_stamp: "Formatted at 2026-09-28 13:08:12 on dist-test-slave-p1lc"
server_key: "a5ec963a8bf3b37b428af19649a57f82"
server_key_iv: "be31b9045b862cec7c11bd682da754ad"
server_key_version: "encryptionkey@0"
I20260928 13:08:12.190148 11931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "88ed2c48bccf409a8440ec3b495a2642"
format_stamp: "Formatted at 2026-09-28 13:08:12 on dist-test-slave-p1lc"
server_key: "a5ec963a8bf3b37b428af19649a57f82"
server_key_iv: "be31b9045b862cec7c11bd682da754ad"
server_key_version: "encryptionkey@0"
I20260928 13:08:12.191526 11931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:08:12.192536 11946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:12.192754 11931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:12.192864 11931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "88ed2c48bccf409a8440ec3b495a2642"
format_stamp: "Formatted at 2026-09-28 13:08:12 on dist-test-slave-p1lc"
server_key: "a5ec963a8bf3b37b428af19649a57f82"
server_key_iv: "be31b9045b862cec7c11bd682da754ad"
server_key_version: "encryptionkey@0"
I20260928 13:08:12.193011 11931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:12.202267 11931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:12.204660 11931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:12.204826 11931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:12.205090 11931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:12.205399 11931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:12.205451 11931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:12.205485 11931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:12.205531 11931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:12.211696 11931 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:33791
I20260928 13:08:12.211735 12059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:33791 every 8 connection(s)
I20260928 13:08:12.212085 11931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:08:12.217134 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 11931
I20260928 13:08:12.217239 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:08:12.217425 8546 external_mini_cluster.cc:1428] Setting key 8fc6bc10a1d9995168a0dbbc638f55a8
Sep 28 13:08:12 dist-test-slave-p1lc krb5kdc[11556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600892, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:12.222609 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:12.212907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:54633 (local address 127.8.88.190:36699)
0928 13:08:12.212998 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:12.213000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:12.213150 (+ 150us) server_negotiation.cc:407] Connection header received
0928 13:08:12.213698 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:12.213699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:12.213708 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:12.213770 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:12.214742 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:12.215229 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:12.216338 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:12.216607 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:12.216673 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:12.219445 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:12.219446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:12.219447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:12.219453 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:12.221021 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:12.221466 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:12.221467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:12.221468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:12.221556 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:12.221868 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:12.221869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:12.221870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:12.221977 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:12.222040 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:12.222445 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 13:08:12.222542 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 13:08:12.222838 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:12.212832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:36699 (local address 127.8.88.131:54633)
0928 13:08:12.213034 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:12.213077 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:12.213606 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:12.213877 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:12.213880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:12.214464 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:12.214659 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:12.214663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:12.215316 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:12.215317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:12.216200 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:12.216203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:12.216694 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:12.216695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:12.216804 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:12.217604 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:12.217619 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:12.219280 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:12.221104 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:12.221110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:12.221112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:12.221395 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:12.221643 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:12.221644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:12.221655 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:12.221796 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:12.222091 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:12.222098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:12.222231 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 13:08:12.222427 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 13:08:12.222591 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":129,"threads_started":1}
I20260928 13:08:12.223413 12060 heartbeater.cc:347] Connected to a master server at 127.8.88.190:36699
I20260928 13:08:12.223505 12060 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:12.223718 12060 heartbeater.cc:511] Master 127.8.88.190:36699 requested a full tablet report, sending...
I20260928 13:08:12.224141 11601 ts_manager.cc:194] Registered new tserver with Master: 88ed2c48bccf409a8440ec3b495a2642 (127.8.88.131:33791)
I20260928 13:08:12.224546 11601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:54633
I20260928 13:08:12.229207 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 13:08:12 dist-test-slave-p1lc krb5kdc[11556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600892, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 13:08:12 dist-test-slave-p1lc krb5kdc[11556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600892, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:12.253855 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:12.245726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.8.88.190:36699)
0928 13:08:12.245805 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:12.245808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:12.245933 (+ 125us) server_negotiation.cc:407] Connection header received
0928 13:08:12.245985 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:12.245986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:12.245998 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:12.246056 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:12.246720 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:12.247216 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:12.248046 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:12.248273 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:12.248333 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:12.250918 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:12.250919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:12.250920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:12.250930 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:12.252462 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:12.252842 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:12.252843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:12.252843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:12.252971 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:12.253213 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:12.253214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:12.253214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:12.253373 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:12.253514 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:12.253689 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 13:08:12.253758 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 13:08:12.254187 11601 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40592
I20260928 13:08:12.254760 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 11663
I20260928 13:08:12.259493 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 11797
I20260928 13:08:12.264256 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 11931
I20260928 13:08:12.268626 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 11569
2026-09-28T13:08:12Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1903 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:08:12 dist-test-slave-p1lc krb5kdc[12072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:08:12 dist-test-slave-p1lc krb5kdc[12072](info): set up 2 sockets
Sep 28 13:08:12 dist-test-slave-p1lc krb5kdc[12072](info): commencing operation
krb5kdc: starting...
W20260928 13:08:13.353185 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:08:13 dist-test-slave-p1lc krb5kdc[12072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:08:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:08:13.480696 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46303
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:13.563076 12085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:13.563320 12085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:13.563345 12085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:13.564677 12085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:13.564735 12085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:13.564750 12085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:13.564764 12085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:13.564776 12085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:13.566533 12085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:13.566833 12085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:13.567112 12085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:13.572386 12093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:13.572386 12090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:13.572422 12091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:13.572646 12085 server_base.cc:1034] running on GCE node
I20260928 13:08:13.573060 12085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:13.573474 12085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:13.574681 12085 hybrid_clock.cc:648] HybridClock initialized: now 1790600893574665 us; error 57 us; skew 500 ppm
Sep 28 13:08:13 dist-test-slave-p1lc krb5kdc[12072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600893, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:13.577677 12085 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:08:13.578263 12085 webserver.cc:460] Webserver started at http://127.8.88.190:33591/ using document root <none> and password file <none>
I20260928 13:08:13.578528 12085 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:13.578608 12085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:13.578780 12085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:13.580046 12085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "ef367875932b44649e3ba648bcbbf07c"
format_stamp: "Formatted at 2026-09-28 13:08:13 on dist-test-slave-p1lc"
server_key: "0ac4e42e0d7b159e08ab1fbcdbabe56a"
server_key_iv: "6f4460c756e0e34aff5cb297013a24b0"
server_key_version: "encryptionkey@0"
I20260928 13:08:13.580489 12085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "ef367875932b44649e3ba648bcbbf07c"
format_stamp: "Formatted at 2026-09-28 13:08:13 on dist-test-slave-p1lc"
server_key: "0ac4e42e0d7b159e08ab1fbcdbabe56a"
server_key_iv: "6f4460c756e0e34aff5cb297013a24b0"
server_key_version: "encryptionkey@0"
I20260928 13:08:13.582155 12085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 13:08:13.583109 12100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:13.583331 12085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:13.583464 12085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "ef367875932b44649e3ba648bcbbf07c"
format_stamp: "Formatted at 2026-09-28 13:08:13 on dist-test-slave-p1lc"
server_key: "0ac4e42e0d7b159e08ab1fbcdbabe56a"
server_key_iv: "6f4460c756e0e34aff5cb297013a24b0"
server_key_version: "encryptionkey@0"
I20260928 13:08:13.583568 12085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:13.595508 12085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:13.598013 12085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:13.598206 12085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:13.602552 12085 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:46303
I20260928 13:08:13.602586 12152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:46303 every 8 connection(s)
I20260928 13:08:13.603133 12085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:13.604036 12153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:13.606031 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 12085
I20260928 13:08:13.606134 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:13.606312 8546 external_mini_cluster.cc:1428] Setting key 20eece0427513fb422813596f181cf40
I20260928 13:08:13.606937 12153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c: Bootstrap starting.
I20260928 13:08:13.607560 12153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:13.607880 12153 log.cc:824] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:13.608580 12153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c: No bootstrap required, opened a new log
I20260928 13:08:13.610493 12153 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef367875932b44649e3ba648bcbbf07c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46303 } }
I20260928 13:08:13.610661 12153 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:13.610706 12153 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef367875932b44649e3ba648bcbbf07c, State: Initialized, Role: FOLLOWER
I20260928 13:08:13.610862 12153 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef367875932b44649e3ba648bcbbf07c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46303 } }
I20260928 13:08:13.610957 12153 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:13.611001 12153 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:13.611055 12153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:13.611614 12153 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef367875932b44649e3ba648bcbbf07c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46303 } }
I20260928 13:08:13.611733 12153 leader_election.cc:302] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef367875932b44649e3ba648bcbbf07c; no voters:
I20260928 13:08:13.612198 12153 leader_election.cc:288] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:13.612277 12158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:13.612496 12158 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [term 1 LEADER]: Becoming Leader. State: Replica: ef367875932b44649e3ba648bcbbf07c, State: Running, Role: LEADER
I20260928 13:08:13.612509 12153 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:13.612665 12158 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef367875932b44649e3ba648bcbbf07c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46303 } }
I20260928 13:08:13.613770 12159 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef367875932b44649e3ba648bcbbf07c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef367875932b44649e3ba648bcbbf07c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46303 } } }
I20260928 13:08:13.613859 12159 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [sys.catalog]: This master's current role is: LEADER
Sep 28 13:08:13 dist-test-slave-p1lc krb5kdc[12072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:13.614169 12160 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef367875932b44649e3ba648bcbbf07c. Latest consensus state: current_term: 1 leader_uuid: "ef367875932b44649e3ba648bcbbf07c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef367875932b44649e3ba648bcbbf07c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46303 } } }
I20260928 13:08:13.614290 12160 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c [sys.catalog]: This master's current role is: LEADER
W20260928 13:08:13.614552 12174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:08:13.614607 12174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:08:13.614732 12172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:13.614928 12172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:13.616636 12172 catalog_manager.cc:1347] Generated new cluster ID: 4c7bea8c2e3143dab703ccc88730936b
I20260928 13:08:13.616695 12172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:08:13.618526 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:13.606868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53082 (local address 127.8.88.190:46303)
0928 13:08:13.607046 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:13.607057 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:13.607094 (+ 37us) server_negotiation.cc:407] Connection header received
0928 13:08:13.607511 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:13.607518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:13.607565 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:13.607726 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:13.608540 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:13.609786 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:13.610698 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:13.611409 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:13.611543 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:13.614670 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:13.614674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:13.614675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:13.614690 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:13.616923 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:13.617293 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:13.617296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:13.617297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:13.617398 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:13.617623 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:13.617625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:13.617625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:13.617879 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:13.618077 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:13.618156 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 13:08:13.618304 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260928 13:08:13.622498 12172 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:13.623041 12172 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:13.625803 12172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef367875932b44649e3ba648bcbbf07c: Generated new TSK 0
I20260928 13:08:13.625977 12172 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:08:13.656746 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46303
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:13.742713 12179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:13.742965 12179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:13.743010 12179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:13.744272 12179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:13.744342 12179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:13.744392 12179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:08:13.746104 12179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:13.746404 12179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:13.746708 12179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:13.752825 12187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:13.752830 12185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:13.752918 12184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:13.753623 12179 server_base.cc:1034] running on GCE node
I20260928 13:08:13.753818 12179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:13.754069 12179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:13.755357 12179 hybrid_clock.cc:648] HybridClock initialized: now 1790600893755333 us; error 44 us; skew 500 ppm
Sep 28 13:08:13 dist-test-slave-p1lc krb5kdc[12072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600893, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:13.758241 12179 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:08:13.758739 12179 webserver.cc:460] Webserver started at http://127.8.88.129:44799/ using document root <none> and password file <none>
I20260928 13:08:13.758950 12179 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:13.758996 12179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:13.759089 12179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:13.760129 12179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "0129863dc0864c158d3e2f14c280136c"
format_stamp: "Formatted at 2026-09-28 13:08:13 on dist-test-slave-p1lc"
server_key: "511d70e5fb420469022586cc4e1159b2"
server_key_iv: "1e8e0ebce16a2d1a0d7775e6bda0f772"
server_key_version: "encryptionkey@0"
I20260928 13:08:13.760440 12179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "0129863dc0864c158d3e2f14c280136c"
format_stamp: "Formatted at 2026-09-28 13:08:13 on dist-test-slave-p1lc"
server_key: "511d70e5fb420469022586cc4e1159b2"
server_key_iv: "1e8e0ebce16a2d1a0d7775e6bda0f772"
server_key_version: "encryptionkey@0"
I20260928 13:08:13.761950 12179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:08:13.762810 12194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:13.763062 12179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:13.763142 12179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "0129863dc0864c158d3e2f14c280136c"
format_stamp: "Formatted at 2026-09-28 13:08:13 on dist-test-slave-p1lc"
server_key: "511d70e5fb420469022586cc4e1159b2"
server_key_iv: "1e8e0ebce16a2d1a0d7775e6bda0f772"
server_key_version: "encryptionkey@0"
I20260928 13:08:13.763200 12179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:13.806437 12179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:13.809031 12179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:13.809159 12179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:13.809365 12179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:13.809719 12179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:13.809757 12179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:13.809782 12179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:13.809832 12179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:13.816210 12179 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:38911
I20260928 13:08:13.816262 12307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:38911 every 8 connection(s)
I20260928 13:08:13.816596 12179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:08:13.823048 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 12179
I20260928 13:08:13.823190 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
Sep 28 13:08:13 dist-test-slave-p1lc krb5kdc[12072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600893, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:13.823382 8546 external_mini_cluster.cc:1428] Setting key 7b375acfd1682e43280face6643b7398
I20260928 13:08:13.827569 12310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:13.817393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46303 (local address 127.8.88.129:42023)
0928 13:08:13.817631 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:13.817670 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:13.818652 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:13.818893 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:13.818896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:13.819159 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:13.819362 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:13.819365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:13.820166 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:13.820167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:13.821121 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:13.821123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:13.821506 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:13.821507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:13.821606 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:13.822248 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:13.822257 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:13.823867 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:13.825922 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:13.825931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:13.825934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:13.826287 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:13.826543 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:13.826544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:13.826544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:13.826699 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:13.827033 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:13.827039 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:13.827165 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 13:08:13.827257 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 13:08:13.827389 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":164,"threads_started":1}
I20260928 13:08:13.827584 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:13.817430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:42023 (local address 127.8.88.190:46303)
0928 13:08:13.817519 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:13.817520 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:13.817762 (+ 242us) server_negotiation.cc:407] Connection header received
0928 13:08:13.818764 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:13.818766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:13.818776 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:13.818834 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:13.819530 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:13.820067 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:13.821215 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:13.821418 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:13.821489 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:13.823946 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:13.823948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:13.823948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:13.823956 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:13.825820 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:13.826371 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:13.826372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:13.826372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:13.826426 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:13.826775 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:13.826776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:13.826776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:13.826959 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:13.827030 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:13.827301 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 13:08:13.827398 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 13:08:13.828172 12308 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46303
I20260928 13:08:13.828305 12308 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:13.828526 12308 heartbeater.cc:511] Master 127.8.88.190:46303 requested a full tablet report, sending...
I20260928 13:08:13.829185 12117 ts_manager.cc:194] Registered new tserver with Master: 0129863dc0864c158d3e2f14c280136c (127.8.88.129:38911)
I20260928 13:08:13.830224 12117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:42023
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:08:13.852793 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46303
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:13.936635 12313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:13.936969 12313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:13.937001 12313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:13.938472 12313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:13.938524 12313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:13.938553 12313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:08:13.940155 12313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:13.940440 12313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:13.940698 12313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:13.946033 12319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:13.946081 12313 server_base.cc:1034] running on GCE node
W20260928 13:08:13.946017 12318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:13.946017 12321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:13.946617 12313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:13.946890 12313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:13.948062 12313 hybrid_clock.cc:648] HybridClock initialized: now 1790600893948039 us; error 41 us; skew 500 ppm
Sep 28 13:08:13 dist-test-slave-p1lc krb5kdc[12072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600893, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:13.950867 12313 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:08:13.951378 12313 webserver.cc:460] Webserver started at http://127.8.88.130:40263/ using document root <none> and password file <none>
I20260928 13:08:13.951637 12313 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:13.951709 12313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:13.951889 12313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:13.953094 12313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "c953f42c8187417c88bcf8d09c082650"
format_stamp: "Formatted at 2026-09-28 13:08:13 on dist-test-slave-p1lc"
server_key: "ce1751835d2c1035486fb0f133f70731"
server_key_iv: "7bec4b955e8c7a575452a0a48f38d365"
server_key_version: "encryptionkey@0"
I20260928 13:08:13.953462 12313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "c953f42c8187417c88bcf8d09c082650"
format_stamp: "Formatted at 2026-09-28 13:08:13 on dist-test-slave-p1lc"
server_key: "ce1751835d2c1035486fb0f133f70731"
server_key_iv: "7bec4b955e8c7a575452a0a48f38d365"
server_key_version: "encryptionkey@0"
I20260928 13:08:13.954991 12313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:08:13.955986 12328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:13.956218 12313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:13.956332 12313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "c953f42c8187417c88bcf8d09c082650"
format_stamp: "Formatted at 2026-09-28 13:08:13 on dist-test-slave-p1lc"
server_key: "ce1751835d2c1035486fb0f133f70731"
server_key_iv: "7bec4b955e8c7a575452a0a48f38d365"
server_key_version: "encryptionkey@0"
I20260928 13:08:13.956403 12313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:13.969919 12313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:13.972314 12313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:13.972492 12313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:13.972739 12313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:13.973088 12313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:13.973145 12313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:13.973181 12313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:13.973208 12313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:13.979166 12313 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:44053
I20260928 13:08:13.979210 12441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:44053 every 8 connection(s)
I20260928 13:08:13.979548 12313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
Sep 28 13:08:13 dist-test-slave-p1lc krb5kdc[12072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600893, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:13.988523 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 12313
I20260928 13:08:13.988646 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:08:13.988853 8546 external_mini_cluster.cc:1428] Setting key e43d7ba977063a1f62459adb19dd2d1b
I20260928 13:08:13.990906 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:13.980282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:44459 (local address 127.8.88.190:46303)
0928 13:08:13.980327 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:13.980330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:13.980585 (+ 255us) server_negotiation.cc:407] Connection header received
0928 13:08:13.981643 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:13.981645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:13.981658 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:13.981712 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:13.982373 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:13.982895 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:13.983945 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:13.984160 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:13.984249 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:13.986883 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:13.986885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:13.986886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:13.986901 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:13.988999 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:13.989678 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:13.989680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:13.989680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:13.989770 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:13.990070 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:13.990071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:13.990072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:13.990235 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:13.990320 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:13.990656 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 13:08:13.990757 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:08:13.991021 12444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:13.980259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46303 (local address 127.8.88.130:44459)
0928 13:08:13.980456 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:13.980495 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:13.981524 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:13.981801 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:13.981805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:13.982071 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:13.982271 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:13.982276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:13.982997 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:13.982998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:13.983845 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:13.983847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:13.984264 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:13.984265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:13.984405 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:13.985051 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:13.985061 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:13.986767 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:13.989128 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:13.989139 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:13.989142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:13.989546 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:13.989847 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:13.989847 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:13.989848 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:13.990000 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:13.990337 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:13.990345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:13.990503 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 13:08:13.990609 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 13:08:13.990761 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260928 13:08:13.991698 12442 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46303
I20260928 13:08:13.991802 12442 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:13.992022 12442 heartbeater.cc:511] Master 127.8.88.190:46303 requested a full tablet report, sending...
I20260928 13:08:13.992502 12117 ts_manager.cc:194] Registered new tserver with Master: c953f42c8187417c88bcf8d09c082650 (127.8.88.130:44053)
I20260928 13:08:13.992969 12117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:44459
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:08:14.017395 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46303
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:36631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:14.102665 12447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:14.102896 12447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:14.102923 12447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:14.104194 12447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:14.104239 12447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:14.104269 12447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:08:14.106060 12447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:36631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:14.106281 12447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:14.106571 12447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:08:14.112283 12447 server_base.cc:1034] running on GCE node
W20260928 13:08:14.112293 12452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:14.112300 12453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:14.112432 12455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:14.112706 12447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:14.113050 12447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:14.114229 12447 hybrid_clock.cc:648] HybridClock initialized: now 1790600894114209 us; error 39 us; skew 500 ppm
Sep 28 13:08:14 dist-test-slave-p1lc krb5kdc[12072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600894, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:14.116990 12447 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:08:14.117475 12447 webserver.cc:460] Webserver started at http://127.8.88.131:39565/ using document root <none> and password file <none>
I20260928 13:08:14.117740 12447 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:14.117812 12447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:14.117977 12447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:14.119216 12447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "cfff0c1531ce4721b92b1579c17aa1bf"
format_stamp: "Formatted at 2026-09-28 13:08:14 on dist-test-slave-p1lc"
server_key: "514da746d28ea603f1014b257f90f78d"
server_key_iv: "182b7f3ed97b50f9fc396f4b84bd6210"
server_key_version: "encryptionkey@0"
I20260928 13:08:14.119594 12447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "cfff0c1531ce4721b92b1579c17aa1bf"
format_stamp: "Formatted at 2026-09-28 13:08:14 on dist-test-slave-p1lc"
server_key: "514da746d28ea603f1014b257f90f78d"
server_key_iv: "182b7f3ed97b50f9fc396f4b84bd6210"
server_key_version: "encryptionkey@0"
I20260928 13:08:14.121106 12447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:08:14.122066 12462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:14.122314 12447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:14.122416 12447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "cfff0c1531ce4721b92b1579c17aa1bf"
format_stamp: "Formatted at 2026-09-28 13:08:14 on dist-test-slave-p1lc"
server_key: "514da746d28ea603f1014b257f90f78d"
server_key_iv: "182b7f3ed97b50f9fc396f4b84bd6210"
server_key_version: "encryptionkey@0"
I20260928 13:08:14.122478 12447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:14.133270 12447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:14.135752 12447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:14.135885 12447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:14.136091 12447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:14.136449 12447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:14.136487 12447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:14.136510 12447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:14.136560 12447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:14.143123 12447 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:38665
I20260928 13:08:14.143163 12575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:38665 every 8 connection(s)
I20260928 13:08:14.143502 12447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
Sep 28 13:08:14 dist-test-slave-p1lc krb5kdc[12072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600894, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:14.152953 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 12447
I20260928 13:08:14.153059 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:08:14.153249 8546 external_mini_cluster.cc:1428] Setting key 7b678d6cf8a48c29db2b610f55badda7
I20260928 13:08:14.153980 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:14.144258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:34003 (local address 127.8.88.190:46303)
0928 13:08:14.144321 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:14.144322 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:14.144505 (+ 183us) server_negotiation.cc:407] Connection header received
0928 13:08:14.145574 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:14.145575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:14.145587 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:14.145670 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:14.146308 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:14.146801 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:14.147888 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:14.148085 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:14.148143 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:14.150732 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:14.150734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:14.150734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:14.150742 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:14.152212 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:14.152679 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:14.152680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:14.152680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:14.152742 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:14.153102 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:14.153104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:14.153104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:14.153246 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:14.153437 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:14.153776 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 13:08:14.153876 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 13:08:14.154281 12578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:14.144227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46303 (local address 127.8.88.131:34003)
0928 13:08:14.144385 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:14.144429 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:14.145453 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:14.145750 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:14.145754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:14.146019 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:14.146217 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:14.146221 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:14.146898 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:14.146898 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:14.147805 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:14.147808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:14.148186 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:14.148187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:14.148322 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:14.149082 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:14.149091 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:14.150636 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:14.152303 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:14.152310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:14.152311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:14.152583 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:14.152825 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:14.152826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:14.152827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:14.153020 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:14.153458 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:14.153463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:14.153640 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 13:08:14.153758 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 13:08:14.153971 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260928 13:08:14.154728 12576 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46303
I20260928 13:08:14.154850 12576 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:14.155051 12576 heartbeater.cc:511] Master 127.8.88.190:46303 requested a full tablet report, sending...
I20260928 13:08:14.155560 12117 ts_manager.cc:194] Registered new tserver with Master: cfff0c1531ce4721b92b1579c17aa1bf (127.8.88.131:38665)
I20260928 13:08:14.155930 12117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:34003
I20260928 13:08:14.165378 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 13:08:14 dist-test-slave-p1lc krb5kdc[12072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 13:08:14 dist-test-slave-p1lc krb5kdc[12072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.129@KRBTEST.COM
I20260928 13:08:14.189630 12583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:14.181045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.8.88.129:38911)
0928 13:08:14.181208 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:14.181213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:14.181231 (+ 18us) server_negotiation.cc:407] Connection header received
0928 13:08:14.181259 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:14.181261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:14.181275 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:14.181349 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:14.182028 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:14.182916 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:14.183701 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:14.184026 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:14.184086 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:14.186490 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:14.186501 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:14.186501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:14.186513 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:14.188151 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:14.188579 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:14.188590 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:14.188590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:14.188669 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:14.188951 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:14.188952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:14.188952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:14.189111 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:14.189381 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:14.189448 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 13:08:14.189529 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
Sep 28 13:08:14 dist-test-slave-p1lc krb5kdc[12072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:14.198079 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:14.190967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53090 (local address 127.8.88.190:46303)
0928 13:08:14.191012 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:14.191014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:14.191101 (+ 87us) server_negotiation.cc:407] Connection header received
0928 13:08:14.191165 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:14.191166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:14.191176 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:14.191220 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:14.191721 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:14.192220 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:14.193165 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:14.193399 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:14.193485 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:14.195695 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:14.195697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:14.195697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:14.195705 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:14.196907 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:14.197229 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:14.197230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:14.197231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:14.197295 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:14.197501 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:14.197502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:14.197502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:14.197655 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:14.197769 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:14.197863 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 13:08:14.197977 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 13:08:14.198324 12117 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53090
I20260928 13:08:14.198653 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 12179
I20260928 13:08:14.203585 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 12313
I20260928 13:08:14.207800 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 12447
I20260928 13:08:14.212453 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 12085
2026-09-28T13:08:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1943 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:08:14 dist-test-slave-p1lc krb5kdc[12588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:08:14 dist-test-slave-p1lc krb5kdc[12588](info): set up 2 sockets
Sep 28 13:08:14 dist-test-slave-p1lc krb5kdc[12588](info): commencing operation
krb5kdc: starting...
W20260928 13:08:15.297205 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:08:15 dist-test-slave-p1lc krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:08:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:08:15.424557 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46691
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:15.507591 12601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:15.507861 12601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:15.508009 12601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:15.509454 12601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:15.509538 12601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:15.509564 12601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:15.509585 12601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:15.509615 12601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:15.511272 12601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:15.511574 12601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:15.511814 12601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:15.516736 12609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:15.516765 12606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:15.516762 12607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:15.517069 12601 server_base.cc:1034] running on GCE node
I20260928 13:08:15.517320 12601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:15.517658 12601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:15.518841 12601 hybrid_clock.cc:648] HybridClock initialized: now 1790600895518824 us; error 52 us; skew 500 ppm
Sep 28 13:08:15 dist-test-slave-p1lc krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:15.526679 12601 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:08:15.527096 12601 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:15.527182 12601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:15.527340 12601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:15.528491 12601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "2d48e75d00ca4f30a150c9c8f1956645"
format_stamp: "Formatted at 2026-09-28 13:08:15 on dist-test-slave-p1lc"
server_key: "6f1ef033ac3bb6d89314d3935f83f13e"
server_key_iv: "73ab096c4c7c63fbe48184b395e00153"
server_key_version: "encryptionkey@0"
I20260928 13:08:15.528930 12601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "2d48e75d00ca4f30a150c9c8f1956645"
format_stamp: "Formatted at 2026-09-28 13:08:15 on dist-test-slave-p1lc"
server_key: "6f1ef033ac3bb6d89314d3935f83f13e"
server_key_iv: "73ab096c4c7c63fbe48184b395e00153"
server_key_version: "encryptionkey@0"
I20260928 13:08:15.530339 12601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:15.531248 12615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:15.531427 12601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:15.531591 12601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "2d48e75d00ca4f30a150c9c8f1956645"
format_stamp: "Formatted at 2026-09-28 13:08:15 on dist-test-slave-p1lc"
server_key: "6f1ef033ac3bb6d89314d3935f83f13e"
server_key_iv: "73ab096c4c7c63fbe48184b395e00153"
server_key_version: "encryptionkey@0"
I20260928 13:08:15.531678 12601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:15.541177 12601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:15.543552 12601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:15.543746 12601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:15.548002 12601 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:46691
I20260928 13:08:15.548024 12667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:46691 every 8 connection(s)
I20260928 13:08:15.548367 12601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:15.549453 12668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:15.549863 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 12601
I20260928 13:08:15.549970 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:15.550187 8546 external_mini_cluster.cc:1428] Setting key 4534da1986119cf2b93ef9b975a9db14
I20260928 13:08:15.552598 12668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645: Bootstrap starting.
I20260928 13:08:15.553368 12668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:15.553659 12668 log.cc:824] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:15.554425 12668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645: No bootstrap required, opened a new log
I20260928 13:08:15.556321 12668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d48e75d00ca4f30a150c9c8f1956645" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46691 } }
I20260928 13:08:15.556494 12668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:15.556525 12668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d48e75d00ca4f30a150c9c8f1956645, State: Initialized, Role: FOLLOWER
I20260928 13:08:15.557034 12668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d48e75d00ca4f30a150c9c8f1956645" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46691 } }
I20260928 13:08:15.557168 12668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:15.557217 12668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:15.557279 12668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 13:08:15 dist-test-slave-p1lc krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:15.557973 12668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d48e75d00ca4f30a150c9c8f1956645" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46691 } }
I20260928 13:08:15.558102 12668 leader_election.cc:302] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d48e75d00ca4f30a150c9c8f1956645; no voters:
I20260928 13:08:15.558307 12668 leader_election.cc:288] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:15.558350 12673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:15.558568 12673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [term 1 LEADER]: Becoming Leader. State: Replica: 2d48e75d00ca4f30a150c9c8f1956645, State: Running, Role: LEADER
I20260928 13:08:15.558673 12668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:15.558687 12673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d48e75d00ca4f30a150c9c8f1956645" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46691 } }
I20260928 13:08:15.559290 12674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d48e75d00ca4f30a150c9c8f1956645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d48e75d00ca4f30a150c9c8f1956645" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46691 } } }
I20260928 13:08:15.559376 12674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:15.559587 12682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:15.560137 12682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:15.561012 12675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d48e75d00ca4f30a150c9c8f1956645. Latest consensus state: current_term: 1 leader_uuid: "2d48e75d00ca4f30a150c9c8f1956645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d48e75d00ca4f30a150c9c8f1956645" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46691 } } }
I20260928 13:08:15.561168 12675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:15.561925 12682 catalog_manager.cc:1347] Generated new cluster ID: df511f086f394b8ba21ca407fb44350f
I20260928 13:08:15.561977 12682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:08:15.562769 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:15.550767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59966 (local address 127.8.88.190:46691)
0928 13:08:15.550965 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:15.550974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:15.551011 (+ 37us) server_negotiation.cc:407] Connection header received
0928 13:08:15.551565 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:15.551574 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:15.551636 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:15.551803 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:15.552433 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:15.553848 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:15.554694 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:15.555137 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:15.555271 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:15.558072 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:15.558077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:15.558077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:15.558094 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:15.560603 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:15.561142 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:15.561146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:15.561146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:15.561248 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:15.561469 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:15.561470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:15.561470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:15.561804 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:15.561978 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:15.562437 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 13:08:15.562572 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260928 13:08:15.571849 12682 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:15.572640 12682 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:15.576812 12682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645: Generated new TSK 0
I20260928 13:08:15.577072 12682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:08:15.611984 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46691
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:15.696903 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:15.697136 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:15.697163 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:15.698433 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:15.698484 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:15.698513 12694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:08:15.700101 12694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:15.700376 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:15.700654 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:15.705994 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:15.706171 12694 server_base.cc:1034] running on GCE node
W20260928 13:08:15.705994 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:15.706014 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:15.706544 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:15.706781 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:15.707933 12694 hybrid_clock.cc:648] HybridClock initialized: now 1790600895707919 us; error 29 us; skew 500 ppm
Sep 28 13:08:15 dist-test-slave-p1lc krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:15.710712 12694 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:08:15.711225 12694 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:15.711306 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:15.711474 12694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:15.712625 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "9e9cb32a0ada49ae860f016ac26bb0f3"
format_stamp: "Formatted at 2026-09-28 13:08:15 on dist-test-slave-p1lc"
server_key: "a627d1b6837dbf4912e701a5e011dbff"
server_key_iv: "7bf81cf29cc3869cd7484ad9cbefbf60"
server_key_version: "encryptionkey@0"
I20260928 13:08:15.713045 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "9e9cb32a0ada49ae860f016ac26bb0f3"
format_stamp: "Formatted at 2026-09-28 13:08:15 on dist-test-slave-p1lc"
server_key: "a627d1b6837dbf4912e701a5e011dbff"
server_key_iv: "7bf81cf29cc3869cd7484ad9cbefbf60"
server_key_version: "encryptionkey@0"
I20260928 13:08:15.714461 12694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:15.715364 12708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:15.715557 12694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:15.715668 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "9e9cb32a0ada49ae860f016ac26bb0f3"
format_stamp: "Formatted at 2026-09-28 13:08:15 on dist-test-slave-p1lc"
server_key: "a627d1b6837dbf4912e701a5e011dbff"
server_key_iv: "7bf81cf29cc3869cd7484ad9cbefbf60"
server_key_version: "encryptionkey@0"
I20260928 13:08:15.715752 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:15.729426 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:15.731829 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:15.732051 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:15.732323 12694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:15.732846 12694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:15.732957 12694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:15.733004 12694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:15.733035 12694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:15.739547 12694 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:37997
I20260928 13:08:15.739625 12821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:37997 every 8 connection(s)
I20260928 13:08:15.739917 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
Sep 28 13:08:15 dist-test-slave-p1lc krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:15.747695 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 12694
I20260928 13:08:15.747828 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:08:15.748044 8546 external_mini_cluster.cc:1428] Setting key 8c0dfb9ca957956338cd2b8fca3bf1d5
I20260928 13:08:15.749692 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:15.740671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:37741 (local address 127.8.88.190:46691)
0928 13:08:15.740717 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:15.740719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:15.740978 (+ 259us) server_negotiation.cc:407] Connection header received
0928 13:08:15.741566 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:15.741567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:15.741578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:15.741652 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:15.742596 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:15.743104 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:15.744212 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:15.744408 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:15.744463 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:15.746646 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:15.746647 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:15.746647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:15.746654 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:15.748147 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:15.748616 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:15.748617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:15.748618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:15.748682 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:15.749011 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:15.749012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:15.749012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:15.749132 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:15.749190 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:15.749554 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 13:08:15.749627 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:08:15.749993 12824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:15.740641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46691 (local address 127.8.88.129:37741)
0928 13:08:15.740808 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:15.740851 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:15.741466 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:15.741709 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:15.741712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:15.741978 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:15.742497 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:15.742503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:15.743192 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:15.743193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:15.744123 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:15.744126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:15.744492 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:15.744493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:15.744596 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:15.745117 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:15.745125 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:15.746562 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:15.748244 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:15.748251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:15.748254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:15.748544 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:15.748745 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:15.748746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:15.748746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:15.748936 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:15.749207 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:15.749214 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:15.749391 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 13:08:15.749515 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 13:08:15.749703 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260928 13:08:15.750702 12822 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46691
I20260928 13:08:15.750819 12822 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:15.750978 12822 heartbeater.cc:511] Master 127.8.88.190:46691 requested a full tablet report, sending...
I20260928 13:08:15.751564 12632 ts_manager.cc:194] Registered new tserver with Master: 9e9cb32a0ada49ae860f016ac26bb0f3 (127.8.88.129:37997)
I20260928 13:08:15.752192 12632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:37741
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:08:15.780573 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46691
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:15.865335 12827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:15.865623 12827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:15.865659 12827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:15.867416 12827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:15.867499 12827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:15.867540 12827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:08:15.869542 12827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:15.869773 12827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:15.869968 12827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:15.875167 12835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:15.875167 12833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:15.875180 12832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:15.875433 12827 server_base.cc:1034] running on GCE node
I20260928 13:08:15.875636 12827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:15.875881 12827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:15.877022 12827 hybrid_clock.cc:648] HybridClock initialized: now 1790600895877007 us; error 30 us; skew 500 ppm
Sep 28 13:08:15 dist-test-slave-p1lc krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:15.879494 12827 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:08:15.879855 12827 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:15.879904 12827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:15.879999 12827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:15.881102 12827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "ff5d0600a32f4dbeb70ea9264dc388c4"
format_stamp: "Formatted at 2026-09-28 13:08:15 on dist-test-slave-p1lc"
server_key: "54295f2753551508d802ad0950ffc830"
server_key_iv: "864c07bf247dea08fd0b2b6c02b3ada9"
server_key_version: "encryptionkey@0"
I20260928 13:08:15.881417 12827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "ff5d0600a32f4dbeb70ea9264dc388c4"
format_stamp: "Formatted at 2026-09-28 13:08:15 on dist-test-slave-p1lc"
server_key: "54295f2753551508d802ad0950ffc830"
server_key_iv: "864c07bf247dea08fd0b2b6c02b3ada9"
server_key_version: "encryptionkey@0"
I20260928 13:08:15.882766 12827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:08:15.883517 12841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:15.883694 12827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:15.883810 12827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "ff5d0600a32f4dbeb70ea9264dc388c4"
format_stamp: "Formatted at 2026-09-28 13:08:15 on dist-test-slave-p1lc"
server_key: "54295f2753551508d802ad0950ffc830"
server_key_iv: "864c07bf247dea08fd0b2b6c02b3ada9"
server_key_version: "encryptionkey@0"
I20260928 13:08:15.883890 12827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:15.896347 12827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:15.898753 12827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:15.898937 12827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:15.899192 12827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:15.899544 12827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:15.899603 12827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:15.899670 12827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:15.899703 12827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:15.906019 12827 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:33661
I20260928 13:08:15.906060 12954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:33661 every 8 connection(s)
I20260928 13:08:15.906365 12827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:08:15.906730 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 12827
I20260928 13:08:15.906847 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:08:15.907126 8546 external_mini_cluster.cc:1428] Setting key 7e03750d797f3f22f22887237ad5e21a
Sep 28 13:08:15 dist-test-slave-p1lc krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600895, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
I20260928 13:08:15.917500 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:15.907257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:56187 (local address 127.8.88.190:46691)
0928 13:08:15.907304 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:15.907306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:15.907589 (+ 283us) server_negotiation.cc:407] Connection header received
0928 13:08:15.908451 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:15.908452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:15.908463 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:15.908556 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:15.909988 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:15.910503 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:15.911607 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:15.911818 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:15.911907 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:15.914332 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:15.914334 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:15.914335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:15.914343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:15.915936 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:15.916375 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:15.916377 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:15.916377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:15.916449 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:15.916736 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:15.916738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:15.916738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:15.916919 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:15.916980 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:15.917299 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 13:08:15.917390 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:08:15.917695 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:15.907250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46691 (local address 127.8.88.130:56187)
0928 13:08:15.907449 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:15.907506 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:15.908305 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:15.908649 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:15.908654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:15.909120 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:15.909875 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:15.909884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:15.910618 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:15.910619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:15.911482 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:15.911484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:15.911920 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:15.911921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:15.912072 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:15.912683 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:15.912692 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:15.914220 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:15.916029 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:15.916039 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:15.916042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:15.916291 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:15.916526 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:15.916527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:15.916527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:15.916663 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:15.916984 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:15.916991 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:15.917151 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 13:08:15.917264 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 13:08:15.917442 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260928 13:08:15.918469 12955 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46691
I20260928 13:08:15.918625 12955 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:15.918807 12955 heartbeater.cc:511] Master 127.8.88.190:46691 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
I20260928 13:08:15.919422 12632 ts_manager.cc:194] Registered new tserver with Master: ff5d0600a32f4dbeb70ea9264dc388c4 (127.8.88.130:33661)
I20260928 13:08:15.920055 12632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:56187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:08:15.937037 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46691
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:16.022439 12960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:16.022675 12960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:16.022702 12960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:16.023850 12960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:16.023888 12960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:16.023916 12960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:08:16.025594 12960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:16.025802 12960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:16.025998 12960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:16.031306 12965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:16.031306 12966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:16.031390 12960 server_base.cc:1034] running on GCE node
W20260928 13:08:16.031304 12968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:16.031800 12960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:16.032141 12960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:16.033316 12960 hybrid_clock.cc:648] HybridClock initialized: now 1790600896033309 us; error 32 us; skew 500 ppm
Sep 28 13:08:16 dist-test-slave-p1lc krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600896, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:16.036242 12960 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:08:16.036677 12960 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:16.036752 12960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:16.036972 12960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:16.038137 12960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "c9c79306e258448aa1f9087c9b50fe0b"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "dbeb00a4a296f116334cfab9fa68ba3c"
server_key_iv: "48714ef216f9b1817be7c82fca3ecc01"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.038568 12960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "c9c79306e258448aa1f9087c9b50fe0b"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "dbeb00a4a296f116334cfab9fa68ba3c"
server_key_iv: "48714ef216f9b1817be7c82fca3ecc01"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.040007 12960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:08:16.040863 12974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:16.041179 12960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:16.041282 12960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "c9c79306e258448aa1f9087c9b50fe0b"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "dbeb00a4a296f116334cfab9fa68ba3c"
server_key_iv: "48714ef216f9b1817be7c82fca3ecc01"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.041373 12960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:16.053781 12960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:16.056138 12960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:16.056351 12960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:16.056684 12960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:16.057107 12960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:16.057164 12960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:16.057210 12960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:16.057235 12960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:16.063819 12960 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:42811
I20260928 13:08:16.063865 13087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:42811 every 8 connection(s)
I20260928 13:08:16.064188 12960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
Sep 28 13:08:16 dist-test-slave-p1lc krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600896, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:16.072496 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 12960
I20260928 13:08:16.072631 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:08:16.072826 8546 external_mini_cluster.cc:1428] Setting key f1c12a8e88bcdb3c1966d093d0429016
I20260928 13:08:16.075605 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.064984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:52851 (local address 127.8.88.190:46691)
0928 13:08:16.065050 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.065052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.065263 (+ 211us) server_negotiation.cc:407] Connection header received
0928 13:08:16.065842 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.065843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.065854 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.065961 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:16.067024 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.067590 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.068664 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.068936 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.069016 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.071328 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:16.071329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:16.071330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:16.071338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:16.073694 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:16.074290 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:16.074291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:16.074292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:16.074388 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:16.074751 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:16.074752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:16.074753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:16.074952 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:16.075028 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.075401 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.075507 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 13:08:16.075773 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.064983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46691 (local address 127.8.88.131:52851)
0928 13:08:16.065133 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.065171 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.065715 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.066040 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.066043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.066310 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:16.066880 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:16.066885 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.067679 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.067680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:16.068563 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:16.068566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.069032 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.069033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:16.069141 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.069663 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:16.069673 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:16.071199 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:16.073808 (+ 2609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:16.073818 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:16.073821 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:16.074169 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:16.074488 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:16.074489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:16.074490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:16.074680 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:16.075031 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:16.075039 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:16.075224 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.075348 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.075538 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260928 13:08:16.076292 13088 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46691
I20260928 13:08:16.076413 13088 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:16.076584 13088 heartbeater.cc:511] Master 127.8.88.190:46691 requested a full tablet report, sending...
I20260928 13:08:16.077106 12632 ts_manager.cc:194] Registered new tserver with Master: c9c79306e258448aa1f9087c9b50fe0b (127.8.88.131:42811)
I20260928 13:08:16.077602 12632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:52851
I20260928 13:08:16.085354 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:08:16.093392 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.086740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59982 (local address 127.8.88.190:46691)
0928 13:08:16.086820 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.086823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.086931 (+ 108us) server_negotiation.cc:407] Connection header received
0928 13:08:16.086981 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.086982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.086992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.087039 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:16.087639 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.088212 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.089094 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.089279 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.089344 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.090682 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:16.090684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:16.090684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:16.090690 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:16.092135 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:16.092520 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:16.092520 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:16.092521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:16.092573 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:16.092740 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:16.092741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:16.092741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:16.092958 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:16.093040 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.093188 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.093255 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 13:08:16.095408 12632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:08:16.097124 12632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:16.104197 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.099718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.8.88.129:37997)
0928 13:08:16.099966 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.099973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.099985 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:08:16.100029 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.100031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.100048 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.100140 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:16.100423 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.101282 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.102969 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.103739 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.103816 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.103887 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.104072 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.104118 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
I20260928 13:08:16.104183 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.099596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:37997 (local address 127.0.0.1:59462)
0928 13:08:16.099723 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.099766 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.099829 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.100250 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.100253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.100255 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:16.100331 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:16.100333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.101516 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.101517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:16.102879 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:16.102882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.103833 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.103834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:16.103916 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.103933 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.104035 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.104096 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":80,"threads_started":1}
I20260928 13:08:16.105233 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.100045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:33661 (local address 127.0.0.1:49448)
0928 13:08:16.101408 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.101416 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.101459 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.101785 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.101785 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.101788 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:16.101840 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:16.101842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.103118 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.103119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:16.103925 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:16.103926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.104968 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.104969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:16.105046 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.105052 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.105099 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.105144 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1346,"thread_start_us":71,"threads_started":1}
I20260928 13:08:16.105331 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.099835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:42811 (local address 127.0.0.1:47516)
0928 13:08:16.100318 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.100329 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.100376 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.101726 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.101727 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.101730 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:16.101785 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:16.101787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.103082 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.103083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:16.104137 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:16.104139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.105084 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.105085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:16.105149 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.105153 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.105218 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.105256 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":73,"threads_started":1}
I20260928 13:08:16.105387 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.099918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.8.88.131:42811)
0928 13:08:16.101557 (+ 1639us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.101561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.101569 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:08:16.101602 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.101604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.101616 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.101713 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:16.101850 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.102974 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.104233 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.104995 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.105068 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.105122 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.105233 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.105278 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1603,"thread_start_us":88,"threads_started":1}
I20260928 13:08:16.105392 12756 tablet_service.cc:1467] Processing CreateTablet for tablet 92feaddf89bc433ca6f295a5b9f71577 (DEFAULT_TABLE table=test-table [id=a4a2555459414d239884962a612b729c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:16.105577 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.100151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49448 (local address 127.8.88.130:33661)
0928 13:08:16.100301 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.100306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.101533 (+ 1227us) server_negotiation.cc:407] Connection header received
0928 13:08:16.101586 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.101588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.101604 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.101694 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:16.101935 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.103006 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.104014 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.104770 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.104839 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.104931 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.105393 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.105504 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260928 13:08:16.105814 12756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92feaddf89bc433ca6f295a5b9f71577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:16.106055 13022 tablet_service.cc:1467] Processing CreateTablet for tablet 92feaddf89bc433ca6f295a5b9f71577 (DEFAULT_TABLE table=test-table [id=a4a2555459414d239884962a612b729c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:16.106179 12889 tablet_service.cc:1467] Processing CreateTablet for tablet 92feaddf89bc433ca6f295a5b9f71577 (DEFAULT_TABLE table=test-table [id=a4a2555459414d239884962a612b729c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:16.106360 13022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92feaddf89bc433ca6f295a5b9f71577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:16.106432 12889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92feaddf89bc433ca6f295a5b9f71577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:16.108995 13105 tablet_bootstrap.cc:492] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4: Bootstrap starting.
I20260928 13:08:16.109804 13105 tablet_bootstrap.cc:654] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:16.110217 13106 tablet_bootstrap.cc:492] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3: Bootstrap starting.
I20260928 13:08:16.110407 13105 log.cc:824] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:16.110838 13106 tablet_bootstrap.cc:654] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:16.111176 13105 tablet_bootstrap.cc:492] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4: No bootstrap required, opened a new log
I20260928 13:08:16.111232 13105 ts_tablet_manager.cc:1397] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:08:16.111370 13106 log.cc:824] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:16.111729 13107 tablet_bootstrap.cc:492] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b: Bootstrap starting.
I20260928 13:08:16.112052 13106 tablet_bootstrap.cc:492] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3: No bootstrap required, opened a new log
I20260928 13:08:16.112104 13106 ts_tablet_manager.cc:1397] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:16.112984 13105 raft_consensus.cc:348] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cb32a0ada49ae860f016ac26bb0f3" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 37997 } } peers { permanent_uuid: "c9c79306e258448aa1f9087c9b50fe0b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42811 } } peers { permanent_uuid: "ff5d0600a32f4dbeb70ea9264dc388c4" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 33661 } }
I20260928 13:08:16.113184 13105 raft_consensus.cc:374] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:16.113234 13105 raft_consensus.cc:729] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5d0600a32f4dbeb70ea9264dc388c4, State: Initialized, Role: FOLLOWER
I20260928 13:08:16.113358 13105 consensus_queue.cc:260] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cb32a0ada49ae860f016ac26bb0f3" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 37997 } } peers { permanent_uuid: "c9c79306e258448aa1f9087c9b50fe0b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42811 } } peers { permanent_uuid: "ff5d0600a32f4dbeb70ea9264dc388c4" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 33661 } }
I20260928 13:08:16.113696 12955 heartbeater.cc:503] Master 127.8.88.190:46691 was elected leader, sending a full tablet report...
I20260928 13:08:16.113679 13105 ts_tablet_manager.cc:1428] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:08:16.113997 13106 raft_consensus.cc:348] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cb32a0ada49ae860f016ac26bb0f3" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 37997 } } peers { permanent_uuid: "c9c79306e258448aa1f9087c9b50fe0b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42811 } } peers { permanent_uuid: "ff5d0600a32f4dbeb70ea9264dc388c4" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 33661 } }
I20260928 13:08:16.114148 13106 raft_consensus.cc:374] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:16.114172 13106 raft_consensus.cc:729] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9cb32a0ada49ae860f016ac26bb0f3, State: Initialized, Role: FOLLOWER
I20260928 13:08:16.114253 13106 consensus_queue.cc:260] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cb32a0ada49ae860f016ac26bb0f3" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 37997 } } peers { permanent_uuid: "c9c79306e258448aa1f9087c9b50fe0b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42811 } } peers { permanent_uuid: "ff5d0600a32f4dbeb70ea9264dc388c4" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 33661 } }
I20260928 13:08:16.114450 13106 ts_tablet_manager.cc:1428] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 13:08:16.114569 12822 heartbeater.cc:503] Master 127.8.88.190:46691 was elected leader, sending a full tablet report...
I20260928 13:08:16.115266 13107 tablet_bootstrap.cc:654] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:16.115752 13107 log.cc:824] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:16.116587 13107 tablet_bootstrap.cc:492] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b: No bootstrap required, opened a new log
I20260928 13:08:16.116653 13107 ts_tablet_manager.cc:1397] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 13:08:16.118130 13107 raft_consensus.cc:348] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cb32a0ada49ae860f016ac26bb0f3" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 37997 } } peers { permanent_uuid: "c9c79306e258448aa1f9087c9b50fe0b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42811 } } peers { permanent_uuid: "ff5d0600a32f4dbeb70ea9264dc388c4" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 33661 } }
I20260928 13:08:16.118361 13107 raft_consensus.cc:374] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:16.118387 13107 raft_consensus.cc:729] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c79306e258448aa1f9087c9b50fe0b, State: Initialized, Role: FOLLOWER
I20260928 13:08:16.118536 13107 consensus_queue.cc:260] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cb32a0ada49ae860f016ac26bb0f3" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 37997 } } peers { permanent_uuid: "c9c79306e258448aa1f9087c9b50fe0b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42811 } } peers { permanent_uuid: "ff5d0600a32f4dbeb70ea9264dc388c4" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 33661 } }
I20260928 13:08:16.118801 13107 ts_tablet_manager.cc:1428] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:08:16.118860 13088 heartbeater.cc:503] Master 127.8.88.190:46691 was elected leader, sending a full tablet report...
W20260928 13:08:16.157131 12956 tablet.cc:2369] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:08:16.227607 13113 raft_consensus.cc:482] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:16.227771 13113 raft_consensus.cc:504] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cb32a0ada49ae860f016ac26bb0f3" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 37997 } } peers { permanent_uuid: "c9c79306e258448aa1f9087c9b50fe0b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42811 } } peers { permanent_uuid: "ff5d0600a32f4dbeb70ea9264dc388c4" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 33661 } }
I20260928 13:08:16.228210 13113 leader_election.cc:288] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e9cb32a0ada49ae860f016ac26bb0f3 (127.8.88.129:37997), ff5d0600a32f4dbeb70ea9264dc388c4 (127.8.88.130:33661)
I20260928 13:08:16.231740 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.228804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:46997 (local address 127.8.88.130:33661)
0928 13:08:16.228892 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.228894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.228901 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:08:16.228926 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.228926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.228937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.228996 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:16.229183 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.229879 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.230992 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.231376 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.231459 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.231469 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.231627 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.231665 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 13:08:16.231755 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.228592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:33661 (local address 127.8.88.131:46997)
0928 13:08:16.228771 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.228781 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.228824 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.229019 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.229021 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.229023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:16.229107 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:16.229110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.230010 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.230011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:16.230865 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:16.230867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.231474 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.231475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:16.231547 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.231550 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.231620 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.231662 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":1,"thread_start_us":120,"threads_started":1}
I20260928 13:08:16.232256 12909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92feaddf89bc433ca6f295a5b9f71577" candidate_uuid: "c9c79306e258448aa1f9087c9b50fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d0600a32f4dbeb70ea9264dc388c4" is_pre_election: true
I20260928 13:08:16.232445 12909 raft_consensus.cc:2454] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9c79306e258448aa1f9087c9b50fe0b in term 0.
I20260928 13:08:16.232533 13090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.228471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:37997 (local address 127.8.88.131:42093)
0928 13:08:16.228580 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.228592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.228642 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.228919 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.228920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.228921 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:16.229029 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:16.229031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.229979 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.229980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:16.231365 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:16.231368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.232278 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.232279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:16.232358 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.232362 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.232420 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.232470 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":26}
I20260928 13:08:16.232710 12978 leader_election.cc:302] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9c79306e258448aa1f9087c9b50fe0b, ff5d0600a32f4dbeb70ea9264dc388c4; no voters:
I20260928 13:08:16.232757 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.228638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:42093 (local address 127.8.88.129:37997)
0928 13:08:16.228695 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.228698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.228712 (+ 14us) server_negotiation.cc:407] Connection header received
0928 13:08:16.228746 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.228747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.228759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.228839 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:16.229122 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.229880 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.231504 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.232185 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.232270 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.232281 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.232645 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.232693 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 13:08:16.232930 13113 raft_consensus.cc:2781] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:16.233016 13113 raft_consensus.cc:482] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:16.233045 13113 raft_consensus.cc:3037] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:16.233073 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92feaddf89bc433ca6f295a5b9f71577" candidate_uuid: "c9c79306e258448aa1f9087c9b50fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9cb32a0ada49ae860f016ac26bb0f3" is_pre_election: true
I20260928 13:08:16.233235 12776 raft_consensus.cc:2454] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9c79306e258448aa1f9087c9b50fe0b in term 0.
I20260928 13:08:16.233952 13113 raft_consensus.cc:504] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cb32a0ada49ae860f016ac26bb0f3" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 37997 } } peers { permanent_uuid: "c9c79306e258448aa1f9087c9b50fe0b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42811 } } peers { permanent_uuid: "ff5d0600a32f4dbeb70ea9264dc388c4" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 33661 } }
I20260928 13:08:16.234095 13113 leader_election.cc:288] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [CANDIDATE]: Term 1 election: Requested vote from peers 9e9cb32a0ada49ae860f016ac26bb0f3 (127.8.88.129:37997), ff5d0600a32f4dbeb70ea9264dc388c4 (127.8.88.130:33661)
I20260928 13:08:16.234308 12909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92feaddf89bc433ca6f295a5b9f71577" candidate_uuid: "c9c79306e258448aa1f9087c9b50fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5d0600a32f4dbeb70ea9264dc388c4"
I20260928 13:08:16.234287 12776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92feaddf89bc433ca6f295a5b9f71577" candidate_uuid: "c9c79306e258448aa1f9087c9b50fe0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9cb32a0ada49ae860f016ac26bb0f3"
I20260928 13:08:16.234373 12909 raft_consensus.cc:3037] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:16.234385 12776 raft_consensus.cc:3037] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:16.235160 12909 raft_consensus.cc:2454] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9c79306e258448aa1f9087c9b50fe0b in term 1.
I20260928 13:08:16.235351 12978 leader_election.cc:302] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9c79306e258448aa1f9087c9b50fe0b, ff5d0600a32f4dbeb70ea9264dc388c4; no voters:
I20260928 13:08:16.235455 13113 raft_consensus.cc:2781] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:16.235561 12776 raft_consensus.cc:2454] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9c79306e258448aa1f9087c9b50fe0b in term 1.
I20260928 13:08:16.235642 13113 raft_consensus.cc:686] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [term 1 LEADER]: Becoming Leader. State: Replica: c9c79306e258448aa1f9087c9b50fe0b, State: Running, Role: LEADER
I20260928 13:08:16.235746 13113 consensus_queue.cc:237] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cb32a0ada49ae860f016ac26bb0f3" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 37997 } } peers { permanent_uuid: "c9c79306e258448aa1f9087c9b50fe0b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42811 } } peers { permanent_uuid: "ff5d0600a32f4dbeb70ea9264dc388c4" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 33661 } }
I20260928 13:08:16.236665 12632 catalog_manager.cc:5495] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b reported cstate change: term changed from 0 to 1, leader changed from <none> to c9c79306e258448aa1f9087c9b50fe0b (127.8.88.131). New cstate: current_term: 1 leader_uuid: "c9c79306e258448aa1f9087c9b50fe0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9cb32a0ada49ae860f016ac26bb0f3" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 37997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9c79306e258448aa1f9087c9b50fe0b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5d0600a32f4dbeb70ea9264dc388c4" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 33661 } health_report { overall_health: UNKNOWN } } }
W20260928 13:08:16.240717 12823 tablet.cc:2369] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:08:16.256459 13102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.253159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.8.88.131:42811)
0928 13:08:16.253243 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.253246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.253257 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:08:16.253306 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.253307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.253334 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.253421 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:16.253606 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.254471 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.255301 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:16.255676 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:16.255757 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:16.255963 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:16.256132 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:16.256227 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.256284 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.256310 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 13:08:16.258630 12909 raft_consensus.cc:1261] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4 [term 1 FOLLOWER]: Refusing update from remote peer c9c79306e258448aa1f9087c9b50fe0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:16.258597 12776 raft_consensus.cc:1261] T 92feaddf89bc433ca6f295a5b9f71577 P 9e9cb32a0ada49ae860f016ac26bb0f3 [term 1 FOLLOWER]: Refusing update from remote peer c9c79306e258448aa1f9087c9b50fe0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:16.259032 13113 consensus_queue.cc:1035] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9cb32a0ada49ae860f016ac26bb0f3" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 37997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:16.259207 13115 consensus_queue.cc:1035] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5d0600a32f4dbeb70ea9264dc388c4" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:16.262157 13118 mvcc.cc:205] Tried to move back new op lower bound from 7334301271071739904 to 7334301270982459392. Current Snapshot: MvccSnapshot[applied={T|T < 7334301271071739904}]
I20260928 13:08:16.265120 12631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:16.265259 12631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:16.266306 12631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d48e75d00ca4f30a150c9c8f1956645: Sending DeleteTablet for 3 replicas of tablet 92feaddf89bc433ca6f295a5b9f71577
I20260928 13:08:16.266908 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet 92feaddf89bc433ca6f295a5b9f71577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:16 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:59462
I20260928 13:08:16.266942 12889 tablet_service.cc:1514] Processing DeleteTablet for tablet 92feaddf89bc433ca6f295a5b9f71577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:16 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:49448
I20260928 13:08:16.266932 13022 tablet_service.cc:1514] Processing DeleteTablet for tablet 92feaddf89bc433ca6f295a5b9f71577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:16 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:47516
I20260928 13:08:16.267235 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 12694
I20260928 13:08:16.273056 13130 tablet_replica.cc:330] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b: stopping tablet replica
I20260928 13:08:16.273216 13130 raft_consensus.cc:2229] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [term 1 LEADER]: Raft consensus shutting down.
W20260928 13:08:16.273427 12619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:37997 (error 108)
I20260928 13:08:16.273488 13130 raft_consensus.cc:2258] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:16.273437 13129 tablet_replica.cc:330] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4: stopping tablet replica
I20260928 13:08:16.273851 13129 raft_consensus.cc:2229] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:16.274077 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 12827
I20260928 13:08:16.274080 13129 raft_consensus.cc:2258] T 92feaddf89bc433ca6f295a5b9f71577 P ff5d0600a32f4dbeb70ea9264dc388c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:16.274997 13130 ts_tablet_manager.cc:1905] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:16.276648 13130 ts_tablet_manager.cc:1918] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:16.276671 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.276488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:37997 (local address 127.0.0.1:59474)
0928 13:08:16.276572 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.276631 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:37997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 13:08:16.276742 13130 log.cc:1196] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/92feaddf89bc433ca6f295a5b9f71577
W20260928 13:08:16.276803 12619 catalog_manager.cc:4584] TS 9e9cb32a0ada49ae860f016ac26bb0f3 (127.8.88.129:37997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 92feaddf89bc433ca6f295a5b9f71577: Network error: Client connection negotiation failed: client connection to 127.8.88.129:37997: connect: Connection refused (error 111)
I20260928 13:08:16.277055 13130 ts_tablet_manager.cc:1939] T 92feaddf89bc433ca6f295a5b9f71577 P c9c79306e258448aa1f9087c9b50fe0b: Deleting consensus metadata
I20260928 13:08:16.277539 12619 catalog_manager.cc:4841] TS c9c79306e258448aa1f9087c9b50fe0b (127.8.88.131:42811): tablet 92feaddf89bc433ca6f295a5b9f71577 (table test-table [id=a4a2555459414d239884962a612b729c]) successfully deleted
I20260928 13:08:16.279078 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 12960
I20260928 13:08:16.279116 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.278965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:33661 (local address 127.0.0.1:49460)
0928 13:08:16.279049 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.279081 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.130:33661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 13:08:16.283586 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 12601
2026-09-28T13:08:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2071 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T13:08:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:16Z Disabled control of system clock
I20260928 13:08:16.312595 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:44651
--webserver_interface=127.8.88.190
--webserver_port=0
--builtin_ntp_servers=127.8.88.148:46625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:44651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 13:08:16.399639 13135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:16.399881 13135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:16.399906 13135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:16.401142 13135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:16.401190 13135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:16.401203 13135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:16.401216 13135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:16.401228 13135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:16.402629 13135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:44651
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:44651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:16.402863 13135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:16.403122 13135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:16.408424 13140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:16.408466 13141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:16.408514 13143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:16.408576 13135 server_base.cc:1034] running on GCE node
I20260928 13:08:16.409065 13135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:16.409391 13135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:16.410593 13135 hybrid_clock.cc:648] HybridClock initialized: now 1790600896410574 us; error 46 us; skew 500 ppm
I20260928 13:08:16.412051 13135 webserver.cc:460] Webserver started at http://127.8.88.190:43829/ using document root <none> and password file <none>
I20260928 13:08:16.412439 13135 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:16.412495 13135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:16.412654 13135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:16.413873 13135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "1d8667da43d541578865e2269d889172"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "967fe673f16572ce4db16b8511f93b9a"
server_key_iv: "306634a075d0db43b06ce8b8bef98603"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.414268 13135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "1d8667da43d541578865e2269d889172"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "967fe673f16572ce4db16b8511f93b9a"
server_key_iv: "306634a075d0db43b06ce8b8bef98603"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.415680 13135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:08:16.416689 13149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:16.417106 13135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:16.417249 13135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "1d8667da43d541578865e2269d889172"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "967fe673f16572ce4db16b8511f93b9a"
server_key_iv: "306634a075d0db43b06ce8b8bef98603"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.417357 13135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:16.421239 13135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:16.421554 13135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:16.421706 13135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:16.425786 13135 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:44651
I20260928 13:08:16.425830 13201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:44651 every 8 connection(s)
I20260928 13:08:16.426210 13135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:16.426941 13202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:16.427572 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 13135
I20260928 13:08:16.427698 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:16.427917 8546 external_mini_cluster.cc:1428] Setting key bc55cc59db4f58e4679b41af3bd311b0
I20260928 13:08:16.430970 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.428625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.8.88.190:44651)
0928 13:08:16.428841 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.428852 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.428940 (+ 88us) server_negotiation.cc:407] Connection header received
0928 13:08:16.429481 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.429490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.429556 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.429723 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:16.430475 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:16.430479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:16.430480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:16.430485 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:16.430536 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:16.430549 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:16.430631 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.430690 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.430753 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260928 13:08:16.431452 13202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172: Bootstrap starting.
I20260928 13:08:16.432176 13202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:16.432476 13202 log.cc:824] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:16.433315 13202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172: No bootstrap required, opened a new log
I20260928 13:08:16.434800 13202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8667da43d541578865e2269d889172" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44651 } }
I20260928 13:08:16.435005 13202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:16.435058 13202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8667da43d541578865e2269d889172, State: Initialized, Role: FOLLOWER
I20260928 13:08:16.435226 13202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8667da43d541578865e2269d889172" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44651 } }
I20260928 13:08:16.435341 13202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:16.435388 13202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:16.435439 13202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:16.436050 13202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8667da43d541578865e2269d889172" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44651 } }
I20260928 13:08:16.436177 13202 leader_election.cc:302] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d8667da43d541578865e2269d889172; no voters:
I20260928 13:08:16.436401 13202 leader_election.cc:288] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:16.436726 13207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:16.436731 13202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:16.436893 13207 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [term 1 LEADER]: Becoming Leader. State: Replica: 1d8667da43d541578865e2269d889172, State: Running, Role: LEADER
I20260928 13:08:16.437003 13207 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8667da43d541578865e2269d889172" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44651 } }
I20260928 13:08:16.437337 13207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d8667da43d541578865e2269d889172. Latest consensus state: current_term: 1 leader_uuid: "1d8667da43d541578865e2269d889172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8667da43d541578865e2269d889172" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44651 } } }
I20260928 13:08:16.437420 13207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:16.437762 13208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d8667da43d541578865e2269d889172" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d8667da43d541578865e2269d889172" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44651 } } }
I20260928 13:08:16.437841 13208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:16.438163 13216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:16.438652 13216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:16.440412 13216 catalog_manager.cc:1347] Generated new cluster ID: a8b1c43d85c541a2aa9a8757295aaad8
I20260928 13:08:16.440460 13216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:08:16.459110 13216 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:16.459352 13216 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:16.466238 13216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172: Generated new TSK 0
I20260928 13:08:16.466480 13216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:08:16.476023 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44651
--builtin_ntp_servers=127.8.88.148:46625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 13:08:16.562394 13226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:16.562642 13226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:16.562668 13226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:16.563977 13226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:16.564028 13226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:16.564057 13226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:08:16.566037 13226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:16.566421 13226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:16.567072 13226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:16.575948 13231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:16.575953 13232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:16.576079 13226 server_base.cc:1034] running on GCE node
W20260928 13:08:16.575948 13234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:16.576452 13226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:16.576773 13226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:16.577982 13226 hybrid_clock.cc:648] HybridClock initialized: now 1790600896577967 us; error 31 us; skew 500 ppm
I20260928 13:08:16.579571 13226 webserver.cc:460] Webserver started at http://127.8.88.129:41389/ using document root <none> and password file <none>
I20260928 13:08:16.579854 13226 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:16.579962 13226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:16.580111 13226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:16.581368 13226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "533fb973711b43d88f78a57c67db866f"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "2d4dc644927a7329583689fa9d0b9625"
server_key_iv: "1b13282dd202440b9bde45b2cacdce5e"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.581750 13226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "533fb973711b43d88f78a57c67db866f"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "2d4dc644927a7329583689fa9d0b9625"
server_key_iv: "1b13282dd202440b9bde45b2cacdce5e"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.583173 13226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:08:16.584152 13240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:16.584354 13226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:16.584471 13226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "533fb973711b43d88f78a57c67db866f"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "2d4dc644927a7329583689fa9d0b9625"
server_key_iv: "1b13282dd202440b9bde45b2cacdce5e"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.584566 13226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:16.588304 13226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:16.588642 13226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:16.588797 13226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:16.589068 13226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:16.589421 13226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:16.589483 13226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:16.589525 13226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:16.589578 13226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:16.595698 13226 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:42823
I20260928 13:08:16.595803 13353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:42823 every 8 connection(s)
I20260928 13:08:16.596169 13226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:08:16.599925 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.597159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:47189 (local address 127.8.88.190:44651)
0928 13:08:16.597242 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.597244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.597537 (+ 293us) server_negotiation.cc:407] Connection header received
0928 13:08:16.598026 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.598027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.598033 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.598095 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:16.599540 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:16.599542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:16.599542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:16.599547 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:16.599563 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:16.599567 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:16.599630 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.599824 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.599840 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 13:08:16.600106 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.597115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:44651 (local address 127.8.88.129:47189)
0928 13:08:16.597343 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.597383 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.597896 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.598290 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.598294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.598303 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:16.599328 (+ 1025us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:08:16.599328 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:16.599362 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:08:16.599395 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:16.599685 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:16.599692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:16.599695 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.599789 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.599854 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":108,"threads_started":1}
I20260928 13:08:16.600754 13354 heartbeater.cc:347] Connected to a master server at 127.8.88.190:44651
I20260928 13:08:16.600925 13354 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:16.601122 13354 heartbeater.cc:511] Master 127.8.88.190:44651 requested a full tablet report, sending...
I20260928 13:08:16.601706 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 13226
I20260928 13:08:16.601786 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:08:16.601809 13166 ts_manager.cc:194] Registered new tserver with Master: 533fb973711b43d88f78a57c67db866f (127.8.88.129:42823)
I20260928 13:08:16.601986 8546 external_mini_cluster.cc:1428] Setting key 0767ec6eb8505903721ca3d0b721bc0f
I20260928 13:08:16.603161 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44651
--builtin_ntp_servers=127.8.88.148:46625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 13:08:16.687569 13357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:16.687791 13357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:16.687815 13357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:16.689185 13357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:16.689235 13357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:16.689263 13357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:08:16.690631 13357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:16.690851 13357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:16.691113 13357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:16.696355 13362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:16.696480 13357 server_base.cc:1034] running on GCE node
W20260928 13:08:16.696611 13365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:16.696611 13363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:16.697012 13357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:16.697283 13357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:16.698470 13357 hybrid_clock.cc:648] HybridClock initialized: now 1790600896698469 us; error 88 us; skew 500 ppm
I20260928 13:08:16.699985 13357 webserver.cc:460] Webserver started at http://127.8.88.130:35511/ using document root <none> and password file <none>
I20260928 13:08:16.700256 13357 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:16.700330 13357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:16.700489 13357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:16.701670 13357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "90e07dd5d84f419fa279fbfa47578255"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "6dd7962c507a1e97dff9888bbcb1859a"
server_key_iv: "f560357b73cd7ca8dd149b0dd7b1f4cb"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.702040 13357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "90e07dd5d84f419fa279fbfa47578255"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "6dd7962c507a1e97dff9888bbcb1859a"
server_key_iv: "f560357b73cd7ca8dd149b0dd7b1f4cb"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.703521 13357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:08:16.704442 13371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:16.704715 13357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:16.704815 13357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "90e07dd5d84f419fa279fbfa47578255"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "6dd7962c507a1e97dff9888bbcb1859a"
server_key_iv: "f560357b73cd7ca8dd149b0dd7b1f4cb"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.704937 13357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:16.708518 13357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:16.708801 13357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:16.708967 13357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:16.709196 13357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:16.709527 13357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:16.709559 13357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:16.709581 13357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:16.709623 13357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:16.715795 13357 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:46623
I20260928 13:08:16.715863 13484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:46623 every 8 connection(s)
I20260928 13:08:16.716193 13357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:08:16.718364 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 13357
I20260928 13:08:16.718489 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:08:16.718722 8546 external_mini_cluster.cc:1428] Setting key 47fdbc067a5034bdf5d3a2a1969bafb0
I20260928 13:08:16.719933 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.717064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:53511 (local address 127.8.88.190:44651)
0928 13:08:16.717137 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.717140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.717419 (+ 279us) server_negotiation.cc:407] Connection header received
0928 13:08:16.717904 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.717905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.717910 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.717997 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:16.719610 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:16.719612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:16.719612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:16.719617 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:16.719638 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:16.719642 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:16.719716 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.719827 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.719852 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 13:08:16.720041 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44651
--builtin_ntp_servers=127.8.88.148:46625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 13:08:16.720163 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.717067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:44651 (local address 127.8.88.130:53511)
0928 13:08:16.717284 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.717319 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.717804 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.718097 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.718101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.718128 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:16.719356 (+ 1228us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:08:16.719356 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:16.719388 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:08:16.719464 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:16.719727 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:16.719733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:16.719737 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.719820 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.719892 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":97,"threads_started":1}
I20260928 13:08:16.720770 13485 heartbeater.cc:347] Connected to a master server at 127.8.88.190:44651
I20260928 13:08:16.720896 13485 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:16.721067 13485 heartbeater.cc:511] Master 127.8.88.190:44651 requested a full tablet report, sending...
I20260928 13:08:16.721424 13166 ts_manager.cc:194] Registered new tserver with Master: 90e07dd5d84f419fa279fbfa47578255 (127.8.88.130:46623)
W20260928 13:08:16.802436 13488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:16.802721 13488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:16.802759 13488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:16.804257 13488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:16.804306 13488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:16.804335 13488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:08:16.805959 13488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:16.806195 13488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:16.806463 13488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:16.812036 13496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:16.811975 13493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:16.812321 13488 server_base.cc:1034] running on GCE node
W20260928 13:08:16.811975 13494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:16.812664 13488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:16.813019 13488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:16.814229 13488 hybrid_clock.cc:648] HybridClock initialized: now 1790600896814217 us; error 37 us; skew 500 ppm
I20260928 13:08:16.815845 13488 webserver.cc:460] Webserver started at http://127.8.88.131:44923/ using document root <none> and password file <none>
I20260928 13:08:16.816149 13488 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:16.816229 13488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:16.816390 13488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:16.817623 13488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "952d65ec411c4b0caabd67ee681d5cec"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "180c0258ea76e7f62129135a4fa6288b"
server_key_iv: "f0f053962ba9279fd1fd73a727d4abc2"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.817998 13488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "952d65ec411c4b0caabd67ee681d5cec"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "180c0258ea76e7f62129135a4fa6288b"
server_key_iv: "f0f053962ba9279fd1fd73a727d4abc2"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.819375 13488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:08:16.820314 13502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:16.820568 13488 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:16.820678 13488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "952d65ec411c4b0caabd67ee681d5cec"
format_stamp: "Formatted at 2026-09-28 13:08:16 on dist-test-slave-p1lc"
server_key: "180c0258ea76e7f62129135a4fa6288b"
server_key_iv: "f0f053962ba9279fd1fd73a727d4abc2"
server_key_version: "encryptionkey@0"
I20260928 13:08:16.820777 13488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:16.824325 13488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:16.824615 13488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:16.824762 13488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:16.825013 13488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:16.825335 13488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:16.825389 13488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:16.825443 13488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:16.825469 13488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:16.831326 13488 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:41483
I20260928 13:08:16.831372 13615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:41483 every 8 connection(s)
I20260928 13:08:16.831754 13488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:08:16.835003 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.832603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:58925 (local address 127.8.88.190:44651)
0928 13:08:16.832674 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.832676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.832846 (+ 170us) server_negotiation.cc:407] Connection header received
0928 13:08:16.833412 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.833414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.833419 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.833493 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:16.834767 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:16.834768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:16.834768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:16.834772 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:16.834788 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:16.834792 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:16.834875 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.834930 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.834944 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 13:08:16.835227 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.832529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:44651 (local address 127.8.88.131:58925)
0928 13:08:16.832715 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.832757 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.833282 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.833551 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.833554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.833562 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:16.834555 (+ 993us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:08:16.834555 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:16.834585 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:08:16.834614 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:16.834871 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:16.834875 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:16.834877 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.834939 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.834996 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260928 13:08:16.835436 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 13488
I20260928 13:08:16.835675 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:08:16.835829 13616 heartbeater.cc:347] Connected to a master server at 127.8.88.190:44651
I20260928 13:08:16.835949 8546 external_mini_cluster.cc:1428] Setting key 32262872c05ccddc0b033970658c02a1
I20260928 13:08:16.835952 13616 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:16.836125 13616 heartbeater.cc:511] Master 127.8.88.190:44651 requested a full tablet report, sending...
I20260928 13:08:16.836526 13166 ts_manager.cc:194] Registered new tserver with Master: 952d65ec411c4b0caabd67ee681d5cec (127.8.88.131:41483)
I20260928 13:08:16.836908 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:08:16.839650 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.838464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.8.88.190:44651)
0928 13:08:16.838547 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.838550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.838626 (+ 76us) server_negotiation.cc:407] Connection header received
0928 13:08:16.838708 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.838709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.838714 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.838784 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:16.839439 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:16.839440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:16.839441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:16.839445 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:16.839460 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:16.839464 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:16.839508 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.839563 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.839578 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 13:08:16.841601 13165 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:08:16.843191 13165 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:16.847581 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.845946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:41483 (local address 127.0.0.1:43464)
0928 13:08:16.846074 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.846085 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.846129 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.846452 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.846458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.846466 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:16.847180 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:08:16.847181 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:16.847197 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:08:16.847220 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:16.847449 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:16.847456 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:16.847460 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.847505 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.847518 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260928 13:08:16.847838 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.846089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43464 (local address 127.8.88.131:41483)
0928 13:08:16.846281 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.846286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.846297 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:08:16.846322 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.846324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.846332 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.846432 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:16.847286 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:16.847293 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:16.847294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:16.847298 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:16.847336 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:16.847360 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:16.847716 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.847746 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.847762 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
I20260928 13:08:16.848285 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.845838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.8.88.129:42823)
0928 13:08:16.846829 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.846834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.846844 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:08:16.846868 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.846870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.846878 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.846957 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:16.847712 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:16.847721 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:16.847722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:16.847727 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:16.847765 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:16.847795 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:16.847893 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.848186 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.848204 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":98,"threads_started":1}
I20260928 13:08:16.848325 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.846266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.8.88.130:46623)
0928 13:08:16.846459 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.846463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.846827 (+ 364us) server_negotiation.cc:407] Connection header received
0928 13:08:16.846861 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.846863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.846872 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.846957 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:16.847887 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:16.847893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:16.847894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:16.847899 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:16.847929 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:16.847955 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:16.848055 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.848232 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.848250 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":106,"threads_started":1}
I20260928 13:08:16.848177 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.846145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:46623 (local address 127.0.0.1:42544)
0928 13:08:16.846752 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.846761 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.846808 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.847304 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.847305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.847307 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:16.847790 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:08:16.847790 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:16.847797 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:08:16.847809 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:16.848053 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:16.848054 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:16.848054 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.848103 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.848116 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":71,"threads_started":1}
I20260928 13:08:16.848542 13550 tablet_service.cc:1467] Processing CreateTablet for tablet d9a794794b4c47e6bd19ef962a32af21 (DEFAULT_TABLE table=test-table [id=6f278f8cfe7d49fc82d7b990431eb948]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:16.848825 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.845708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:42823 (local address 127.0.0.1:33044)
0928 13:08:16.845865 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.845908 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.845969 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.846958 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.846959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.846962 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:16.847380 (+ 418us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:08:16.847381 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:16.847388 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:08:16.847402 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:16.848110 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:16.848111 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:16.848112 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.848173 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.848187 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260928 13:08:16.849033 13550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a794794b4c47e6bd19ef962a32af21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:16.849089 13419 tablet_service.cc:1467] Processing CreateTablet for tablet d9a794794b4c47e6bd19ef962a32af21 (DEFAULT_TABLE table=test-table [id=6f278f8cfe7d49fc82d7b990431eb948]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:16.849320 13419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a794794b4c47e6bd19ef962a32af21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:16.849658 13288 tablet_service.cc:1467] Processing CreateTablet for tablet d9a794794b4c47e6bd19ef962a32af21 (DEFAULT_TABLE table=test-table [id=6f278f8cfe7d49fc82d7b990431eb948]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:16.849967 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a794794b4c47e6bd19ef962a32af21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:16.851761 13630 tablet_bootstrap.cc:492] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255: Bootstrap starting.
I20260928 13:08:16.852384 13631 tablet_bootstrap.cc:492] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec: Bootstrap starting.
I20260928 13:08:16.852520 13630 tablet_bootstrap.cc:654] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:16.852841 13630 log.cc:824] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:16.853041 13631 tablet_bootstrap.cc:654] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:16.853343 13631 log.cc:824] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:16.853406 13632 tablet_bootstrap.cc:492] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f: Bootstrap starting.
I20260928 13:08:16.853600 13630 tablet_bootstrap.cc:492] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255: No bootstrap required, opened a new log
I20260928 13:08:16.853694 13630 ts_tablet_manager.cc:1397] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:16.854001 13632 tablet_bootstrap.cc:654] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:16.854087 13631 tablet_bootstrap.cc:492] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec: No bootstrap required, opened a new log
I20260928 13:08:16.854180 13631 ts_tablet_manager.cc:1397] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 13:08:16.854359 13632 log.cc:824] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:16.855028 13632 tablet_bootstrap.cc:492] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f: No bootstrap required, opened a new log
I20260928 13:08:16.855113 13632 ts_tablet_manager.cc:1397] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:16.856434 13630 raft_consensus.cc:348] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533fb973711b43d88f78a57c67db866f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42823 } } peers { permanent_uuid: "952d65ec411c4b0caabd67ee681d5cec" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 41483 } } peers { permanent_uuid: "90e07dd5d84f419fa279fbfa47578255" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 46623 } }
I20260928 13:08:16.856647 13630 raft_consensus.cc:374] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:16.856688 13630 raft_consensus.cc:729] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e07dd5d84f419fa279fbfa47578255, State: Initialized, Role: FOLLOWER
I20260928 13:08:16.856701 13631 raft_consensus.cc:348] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533fb973711b43d88f78a57c67db866f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42823 } } peers { permanent_uuid: "952d65ec411c4b0caabd67ee681d5cec" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 41483 } } peers { permanent_uuid: "90e07dd5d84f419fa279fbfa47578255" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 46623 } }
I20260928 13:08:16.856899 13631 raft_consensus.cc:374] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:16.856938 13631 raft_consensus.cc:729] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952d65ec411c4b0caabd67ee681d5cec, State: Initialized, Role: FOLLOWER
I20260928 13:08:16.856931 13630 consensus_queue.cc:260] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533fb973711b43d88f78a57c67db866f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42823 } } peers { permanent_uuid: "952d65ec411c4b0caabd67ee681d5cec" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 41483 } } peers { permanent_uuid: "90e07dd5d84f419fa279fbfa47578255" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 46623 } }
I20260928 13:08:16.857069 13631 consensus_queue.cc:260] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533fb973711b43d88f78a57c67db866f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42823 } } peers { permanent_uuid: "952d65ec411c4b0caabd67ee681d5cec" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 41483 } } peers { permanent_uuid: "90e07dd5d84f419fa279fbfa47578255" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 46623 } }
I20260928 13:08:16.857021 13632 raft_consensus.cc:348] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533fb973711b43d88f78a57c67db866f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42823 } } peers { permanent_uuid: "952d65ec411c4b0caabd67ee681d5cec" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 41483 } } peers { permanent_uuid: "90e07dd5d84f419fa279fbfa47578255" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 46623 } }
I20260928 13:08:16.857183 13632 raft_consensus.cc:374] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:16.857228 13632 raft_consensus.cc:729] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533fb973711b43d88f78a57c67db866f, State: Initialized, Role: FOLLOWER
I20260928 13:08:16.857370 13630 ts_tablet_manager.cc:1428] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260928 13:08:16.857370 13631 ts_tablet_manager.cc:1428] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 13:08:16.857410 13616 heartbeater.cc:503] Master 127.8.88.190:44651 was elected leader, sending a full tablet report...
I20260928 13:08:16.857496 13485 heartbeater.cc:503] Master 127.8.88.190:44651 was elected leader, sending a full tablet report...
I20260928 13:08:16.857372 13632 consensus_queue.cc:260] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533fb973711b43d88f78a57c67db866f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42823 } } peers { permanent_uuid: "952d65ec411c4b0caabd67ee681d5cec" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 41483 } } peers { permanent_uuid: "90e07dd5d84f419fa279fbfa47578255" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 46623 } }
I20260928 13:08:16.857944 13632 ts_tablet_manager.cc:1428] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:08:16.857972 13354 heartbeater.cc:503] Master 127.8.88.190:44651 was elected leader, sending a full tablet report...
I20260928 13:08:16.962121 13638 raft_consensus.cc:482] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:16.962288 13638 raft_consensus.cc:504] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533fb973711b43d88f78a57c67db866f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42823 } } peers { permanent_uuid: "952d65ec411c4b0caabd67ee681d5cec" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 41483 } } peers { permanent_uuid: "90e07dd5d84f419fa279fbfa47578255" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 46623 } }
I20260928 13:08:16.962677 13638 leader_election.cc:288] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 952d65ec411c4b0caabd67ee681d5cec (127.8.88.131:41483), 90e07dd5d84f419fa279fbfa47578255 (127.8.88.130:46623)
I20260928 13:08:16.964334 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.962901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:41483 (local address 127.8.88.129:55041)
0928 13:08:16.963000 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.963017 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.963093 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.963323 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.963324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.963326 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:16.963908 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:08:16.963909 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:16.963918 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:08:16.963933 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:16.964161 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:16.964162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:16.964162 (+ 0us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.964241 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.964259 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 13:08:16.964334 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.962989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:55041 (local address 127.8.88.131:41483)
0928 13:08:16.963067 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.963069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.963082 (+ 13us) server_negotiation.cc:407] Connection header received
0928 13:08:16.963186 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.963188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.963195 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.963260 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:16.964038 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:16.964040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:16.964040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:16.964044 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:16.964074 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:16.964080 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:16.964144 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.964246 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.964261 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 13:08:16.964419 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.963210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:34641 (local address 127.8.88.130:46623)
0928 13:08:16.963272 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.963275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.963285 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:08:16.963347 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.963348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.963353 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.963432 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:16.964196 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:16.964197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:16.964197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:16.964201 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:16.964216 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:16.964221 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:16.964290 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.964339 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.964354 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 13:08:16.964433 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.963046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:46623 (local address 127.8.88.129:34641)
0928 13:08:16.963227 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:16.963237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:16.963285 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:16.963437 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:16.963438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:16.963441 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:16.964116 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:08:16.964116 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:16.964123 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:08:16.964137 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:16.964288 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:16.964288 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:16.964289 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 13:08:16.964341 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 13:08:16.964354 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":104,"threads_started":1}
I20260928 13:08:16.964771 13570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a794794b4c47e6bd19ef962a32af21" candidate_uuid: "533fb973711b43d88f78a57c67db866f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952d65ec411c4b0caabd67ee681d5cec" is_pre_election: true
I20260928 13:08:16.964769 13439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a794794b4c47e6bd19ef962a32af21" candidate_uuid: "533fb973711b43d88f78a57c67db866f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90e07dd5d84f419fa279fbfa47578255" is_pre_election: true
I20260928 13:08:16.964948 13439 raft_consensus.cc:2454] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533fb973711b43d88f78a57c67db866f in term 0.
I20260928 13:08:16.964948 13570 raft_consensus.cc:2454] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533fb973711b43d88f78a57c67db866f in term 0.
I20260928 13:08:16.965284 13242 leader_election.cc:302] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533fb973711b43d88f78a57c67db866f, 952d65ec411c4b0caabd67ee681d5cec; no voters:
I20260928 13:08:16.965430 13638 raft_consensus.cc:2781] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:16.965483 13638 raft_consensus.cc:482] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:16.965515 13638 raft_consensus.cc:3037] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:16.966209 13638 raft_consensus.cc:504] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533fb973711b43d88f78a57c67db866f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42823 } } peers { permanent_uuid: "952d65ec411c4b0caabd67ee681d5cec" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 41483 } } peers { permanent_uuid: "90e07dd5d84f419fa279fbfa47578255" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 46623 } }
I20260928 13:08:16.966339 13638 leader_election.cc:288] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [CANDIDATE]: Term 1 election: Requested vote from peers 952d65ec411c4b0caabd67ee681d5cec (127.8.88.131:41483), 90e07dd5d84f419fa279fbfa47578255 (127.8.88.130:46623)
I20260928 13:08:16.966498 13570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a794794b4c47e6bd19ef962a32af21" candidate_uuid: "533fb973711b43d88f78a57c67db866f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952d65ec411c4b0caabd67ee681d5cec"
I20260928 13:08:16.966574 13570 raft_consensus.cc:3037] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:16.966538 13439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a794794b4c47e6bd19ef962a32af21" candidate_uuid: "533fb973711b43d88f78a57c67db866f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90e07dd5d84f419fa279fbfa47578255"
I20260928 13:08:16.966665 13439 raft_consensus.cc:3037] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:16.967312 13570 raft_consensus.cc:2454] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533fb973711b43d88f78a57c67db866f in term 1.
I20260928 13:08:16.967465 13242 leader_election.cc:302] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 533fb973711b43d88f78a57c67db866f, 952d65ec411c4b0caabd67ee681d5cec; no voters:
I20260928 13:08:16.967545 13638 raft_consensus.cc:2781] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:16.967572 13439 raft_consensus.cc:2454] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533fb973711b43d88f78a57c67db866f in term 1.
I20260928 13:08:16.967684 13638 raft_consensus.cc:686] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [term 1 LEADER]: Becoming Leader. State: Replica: 533fb973711b43d88f78a57c67db866f, State: Running, Role: LEADER
W20260928 13:08:16.967741 13486 tablet.cc:2369] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:08:16.967793 13638 consensus_queue.cc:237] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533fb973711b43d88f78a57c67db866f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42823 } } peers { permanent_uuid: "952d65ec411c4b0caabd67ee681d5cec" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 41483 } } peers { permanent_uuid: "90e07dd5d84f419fa279fbfa47578255" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 46623 } }
I20260928 13:08:16.968720 13166 catalog_manager.cc:5495] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f reported cstate change: term changed from 0 to 1, leader changed from <none> to 533fb973711b43d88f78a57c67db866f (127.8.88.129). New cstate: current_term: 1 leader_uuid: "533fb973711b43d88f78a57c67db866f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533fb973711b43d88f78a57c67db866f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952d65ec411c4b0caabd67ee681d5cec" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 41483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90e07dd5d84f419fa279fbfa47578255" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 46623 } health_report { overall_health: UNKNOWN } } }
I20260928 13:08:16.997817 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:16.996711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.8.88.129:42823)
0928 13:08:16.996787 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:16.996789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:16.996796 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:08:16.996833 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:16.996834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:16.996848 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:16.996974 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:16.997564 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:16.997565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:16.997565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:16.997569 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:16.997586 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:16.997590 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:16.997661 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:16.997720 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 13:08:16.997736 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 13:08:16.999749 13439 raft_consensus.cc:1261] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255 [term 1 FOLLOWER]: Refusing update from remote peer 533fb973711b43d88f78a57c67db866f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:16.999749 13570 raft_consensus.cc:1261] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec [term 1 FOLLOWER]: Refusing update from remote peer 533fb973711b43d88f78a57c67db866f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:17.000140 13638 consensus_queue.cc:1035] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e07dd5d84f419fa279fbfa47578255" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 46623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:17.000299 13640 consensus_queue.cc:1035] T d9a794794b4c47e6bd19ef962a32af21 P 533fb973711b43d88f78a57c67db866f [LEADER]: Connected to new peer: Peer: permanent_uuid: "952d65ec411c4b0caabd67ee681d5cec" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 41483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:17.002921 13645 mvcc.cc:205] Tried to move back new op lower bound from 7334301274107985920 to 7334301273980583936. Current Snapshot: MvccSnapshot[applied={T|T < 7334301274107985920}]
I20260928 13:08:17.005677 13164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:17.005797 13164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:17.006626 13164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d8667da43d541578865e2269d889172: Sending DeleteTablet for 3 replicas of tablet d9a794794b4c47e6bd19ef962a32af21
I20260928 13:08:17.006953 13288 tablet_service.cc:1514] Processing DeleteTablet for tablet d9a794794b4c47e6bd19ef962a32af21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:17 UTC) from {username='slave'} at 127.0.0.1:33044
I20260928 13:08:17.007076 13550 tablet_service.cc:1514] Processing DeleteTablet for tablet d9a794794b4c47e6bd19ef962a32af21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:17 UTC) from {username='slave'} at 127.0.0.1:43464
I20260928 13:08:17.007225 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 13226
I20260928 13:08:17.007298 13419 tablet_service.cc:1514] Processing DeleteTablet for tablet d9a794794b4c47e6bd19ef962a32af21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:17 UTC) from {username='slave'} at 127.0.0.1:42544
I20260928 13:08:17.007884 13654 tablet_replica.cc:330] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec: stopping tablet replica
I20260928 13:08:17.007993 13654 raft_consensus.cc:2229] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:17.008105 13654 raft_consensus.cc:2258] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:17.008203 13655 tablet_replica.cc:330] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255: stopping tablet replica
I20260928 13:08:17.008306 13655 raft_consensus.cc:2229] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:17.008533 13655 raft_consensus.cc:2258] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:17.008607 13654 ts_tablet_manager.cc:1905] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:17.008994 13655 ts_tablet_manager.cc:1905] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:17.010639 13654 ts_tablet_manager.cc:1918] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:17.010730 13654 log.cc:1196] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/d9a794794b4c47e6bd19ef962a32af21
I20260928 13:08:17.011101 13654 ts_tablet_manager.cc:1939] T d9a794794b4c47e6bd19ef962a32af21 P 952d65ec411c4b0caabd67ee681d5cec: Deleting consensus metadata
I20260928 13:08:17.011157 13655 ts_tablet_manager.cc:1918] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:17.011242 13655 log.cc:1196] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/d9a794794b4c47e6bd19ef962a32af21
I20260928 13:08:17.011538 13655 ts_tablet_manager.cc:1939] T d9a794794b4c47e6bd19ef962a32af21 P 90e07dd5d84f419fa279fbfa47578255: Deleting consensus metadata
I20260928 13:08:17.011612 13151 catalog_manager.cc:4841] TS 952d65ec411c4b0caabd67ee681d5cec (127.8.88.131:41483): tablet d9a794794b4c47e6bd19ef962a32af21 (table test-table [id=6f278f8cfe7d49fc82d7b990431eb948]) successfully deleted
I20260928 13:08:17.012048 13151 catalog_manager.cc:4841] TS 90e07dd5d84f419fa279fbfa47578255 (127.8.88.130:46623): tablet d9a794794b4c47e6bd19ef962a32af21 (table test-table [id=6f278f8cfe7d49fc82d7b990431eb948]) successfully deleted
W20260928 13:08:17.015735 13151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:42823 (error 108)
I20260928 13:08:17.016288 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 13357
I20260928 13:08:17.016321 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:17.016178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:42823 (local address 127.0.0.1:33068)
0928 13:08:17.016238 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:17.016290 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:42823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 13:08:17.017839 13151 catalog_manager.cc:4584] TS 533fb973711b43d88f78a57c67db866f (127.8.88.129:42823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9a794794b4c47e6bd19ef962a32af21: Network error: Client connection negotiation failed: client connection to 127.8.88.129:42823: connect: Connection refused (error 111)
I20260928 13:08:17.021620 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 13488
I20260928 13:08:17.026791 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 13135
2026-09-28T13:08:17Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (743 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 13:08:17.047847 8546 mini_oidc.cc:152] Starting JWKS server
I20260928 13:08:17.049089 8546 webserver.cc:460] Webserver started at https://127.0.0.1:45239/ using document root <none> and password file <none>
I20260928 13:08:17.049209 8546 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 13:08:17.049418 8546 webserver.cc:460] Webserver started at http://127.0.0.1:45621/ using document root <none> and password file <none>
2026-09-28T13:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:17Z Disabled control of system clock
I20260928 13:08:17.058415 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:41469
--webserver_interface=127.8.88.190
--webserver_port=0
--builtin_ntp_servers=127.8.88.148:45567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:41469
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45239/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/testchainca.cert with env {}
W20260928 13:08:17.145064 13662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:17.145283 13662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:17.145306 13662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:17.146492 13662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:17.146533 13662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:17.146545 13662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:17.146560 13662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:17.146572 13662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 13:08:17.146584 13662 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 13:08:17.146596 13662 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45239/jwks.json
I20260928 13:08:17.148118 13662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:45567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:41469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:41469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45239/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:17.148317 13662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:17.148504 13662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:17.153807 13667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:17.153825 13668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:17.153825 13670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:17.154338 13662 server_base.cc:1034] running on GCE node
I20260928 13:08:17.154636 13662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:17.154937 13662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:17.156097 13662 hybrid_clock.cc:648] HybridClock initialized: now 1790600897156085 us; error 35 us; skew 500 ppm
I20260928 13:08:17.157595 13662 webserver.cc:460] Webserver started at http://127.8.88.190:40653/ using document root <none> and password file <none>
I20260928 13:08:17.157894 13662 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:17.157946 13662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:17.158105 13662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:17.159221 13662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "2035694f0f6c4dd38ec3701fcf470ae5"
format_stamp: "Formatted at 2026-09-28 13:08:17 on dist-test-slave-p1lc"
server_key: "7fac3e1a1b3a4a5aeeaa53b2e0730b6c"
server_key_iv: "e4f0f98c7354b5d84a475ee1407c2e90"
server_key_version: "encryptionkey@0"
I20260928 13:08:17.159606 13662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "2035694f0f6c4dd38ec3701fcf470ae5"
format_stamp: "Formatted at 2026-09-28 13:08:17 on dist-test-slave-p1lc"
server_key: "7fac3e1a1b3a4a5aeeaa53b2e0730b6c"
server_key_iv: "e4f0f98c7354b5d84a475ee1407c2e90"
server_key_version: "encryptionkey@0"
I20260928 13:08:17.161166 13662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 13:08:17.162086 13676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:17.162343 13662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:17.162422 13662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "2035694f0f6c4dd38ec3701fcf470ae5"
format_stamp: "Formatted at 2026-09-28 13:08:17 on dist-test-slave-p1lc"
server_key: "7fac3e1a1b3a4a5aeeaa53b2e0730b6c"
server_key_iv: "e4f0f98c7354b5d84a475ee1407c2e90"
server_key_version: "encryptionkey@0"
I20260928 13:08:17.162533 13662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:17.180006 13662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:17.180352 13662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:17.180526 13662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:17.184775 13662 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:41469
I20260928 13:08:17.184877 13728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:41469 every 8 connection(s)
I20260928 13:08:17.185243 13662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:17.186291 13729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:17.188969 13729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5: Bootstrap starting.
I20260928 13:08:17.189649 13729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:17.190001 13729 log.cc:824] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:17.190923 13729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5: No bootstrap required, opened a new log
I20260928 13:08:17.192636 13729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2035694f0f6c4dd38ec3701fcf470ae5" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 41469 } }
I20260928 13:08:17.192813 13729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:17.192839 13729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2035694f0f6c4dd38ec3701fcf470ae5, State: Initialized, Role: FOLLOWER
I20260928 13:08:17.193084 13729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2035694f0f6c4dd38ec3701fcf470ae5" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 41469 } }
I20260928 13:08:17.193177 13729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:17.193219 13729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:17.193274 13729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:17.193636 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 13662
I20260928 13:08:17.193740 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:17.193919 8546 external_mini_cluster.cc:1428] Setting key 5586143031106070c4807998ca592146
I20260928 13:08:17.194113 13729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2035694f0f6c4dd38ec3701fcf470ae5" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 41469 } }
I20260928 13:08:17.194218 13729 leader_election.cc:302] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2035694f0f6c4dd38ec3701fcf470ae5; no voters:
I20260928 13:08:17.194432 13729 leader_election.cc:288] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:17.194545 13732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:17.194726 13729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:17.194742 13732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [term 1 LEADER]: Becoming Leader. State: Replica: 2035694f0f6c4dd38ec3701fcf470ae5, State: Running, Role: LEADER
I20260928 13:08:17.194958 13732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2035694f0f6c4dd38ec3701fcf470ae5" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 41469 } }
I20260928 13:08:17.195538 13734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2035694f0f6c4dd38ec3701fcf470ae5. Latest consensus state: current_term: 1 leader_uuid: "2035694f0f6c4dd38ec3701fcf470ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2035694f0f6c4dd38ec3701fcf470ae5" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 41469 } } }
I20260928 13:08:17.195569 13736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2035694f0f6c4dd38ec3701fcf470ae5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2035694f0f6c4dd38ec3701fcf470ae5" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 41469 } } }
I20260928 13:08:17.195669 13736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:17.196091 13734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:17.196117 13743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:17.196674 13743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:17.198452 13743 catalog_manager.cc:1347] Generated new cluster ID: b65de69f9cde48518edf945038dedb7a
I20260928 13:08:17.198521 13743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:08:17.200987 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:17.194474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.8.88.190:41469)
0928 13:08:17.194679 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:17.194691 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:17.194971 (+ 280us) server_negotiation.cc:407] Connection header received
0928 13:08:17.195123 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:17.195131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:17.195221 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:17.195384 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:17.196124 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.197535 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.198674 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.199337 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.199464 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:17.200194 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:17.200199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:17.200199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:17.200420 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:17.200469 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:17.200485 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:17.200561 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:17.200631 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 13:08:17.200718 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":88,"threads_started":1}
I20260928 13:08:17.202052 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:08:17.217672 13743 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:17.218242 13743 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:17.228063 13743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2035694f0f6c4dd38ec3701fcf470ae5: Generated new TSK 0
I20260928 13:08:17.228307 13743 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 13:08:17.259297 13759 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 13:08:17.259419 13759 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 13:08:17.259835 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:17.236265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43392 (local address 127.8.88.190:41469)
0928 13:08:17.236346 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:17.236349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:17.236545 (+ 196us) server_negotiation.cc:407] Connection header received
0928 13:08:17.236613 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:17.236614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:17.236625 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:17.236691 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0928 13:08:17.236861 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.237441 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.238605 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.238791 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.238859 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:17.239138 (+ 279us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 13:08:17.259308 (+ 20170us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 13:08:17.259738 (+ 430us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43392: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 13:08:17.259904 13735 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43392: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 13:08:17.260166 13758 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:08:17.236191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:41469 (local address 127.0.0.1:43392)
0928 13:08:17.236420 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:17.236432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:17.236513 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:17.236716 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:17.236717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:17.236719 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 13:08:17.236791 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:17.236794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.237528 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.237529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:17.238490 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:17.238493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.238883 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.238884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:17.238953 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:17.238979 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 13:08:17.259686 (+ 20707us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 13:08:17.259818 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.88.190:41469: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":120,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.88.190:41469: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 13:08:17.260679 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 13662
2026-09-28T13:08:17Z chronyd exiting
I20260928 13:08:17.297230 8546 test_util.cc:175] -----------------------------------------------
I20260928 13:08:17.297334 8546 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790600874459249-8546-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (250 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 13:08:17.298102 8546 mini_oidc.cc:152] Starting JWKS server
I20260928 13:08:17.298900 8546 webserver.cc:460] Webserver started at https://127.0.0.1:40637/ using document root <none> and password file <none>
I20260928 13:08:17.298949 8546 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 13:08:17.299144 8546 webserver.cc:460] Webserver started at http://127.0.0.1:45867/ using document root <none> and password file <none>
2026-09-28T13:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:17Z Disabled control of system clock
I20260928 13:08:17.307130 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:44431
--webserver_interface=127.8.88.190
--webserver_port=0
--builtin_ntp_servers=127.8.88.148:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:44431
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40637/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/testchainca.cert with env {}
W20260928 13:08:17.395027 13765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:17.395257 13765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:17.395282 13765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:17.396481 13765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:17.396525 13765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:17.396538 13765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:17.396550 13765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:17.396562 13765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 13:08:17.396576 13765 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 13:08:17.396588 13765 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40637/jwks.json
I20260928 13:08:17.398169 13765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:44431
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:44431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40637/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:17.398387 13765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:17.398557 13765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:17.403616 13773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:17.403616 13770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:17.403659 13771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:17.403734 13765 server_base.cc:1034] running on GCE node
I20260928 13:08:17.404181 13765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:17.404531 13765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:17.405742 13765 hybrid_clock.cc:648] HybridClock initialized: now 1790600897405726 us; error 69 us; skew 500 ppm
I20260928 13:08:17.407153 13765 webserver.cc:460] Webserver started at http://127.8.88.190:44131/ using document root <none> and password file <none>
I20260928 13:08:17.407416 13765 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:17.407490 13765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:17.407686 13765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:17.409170 13765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "f74c4dd494fb4feca3bde25c6291e68c"
format_stamp: "Formatted at 2026-09-28 13:08:17 on dist-test-slave-p1lc"
server_key: "912eacc892d623c4544c8b484e6b8d02"
server_key_iv: "6ba58cc390c940e3a7519cacaf7923cb"
server_key_version: "encryptionkey@0"
I20260928 13:08:17.409564 13765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "f74c4dd494fb4feca3bde25c6291e68c"
format_stamp: "Formatted at 2026-09-28 13:08:17 on dist-test-slave-p1lc"
server_key: "912eacc892d623c4544c8b484e6b8d02"
server_key_iv: "6ba58cc390c940e3a7519cacaf7923cb"
server_key_version: "encryptionkey@0"
I20260928 13:08:17.410964 13765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:08:17.411899 13779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:17.412148 13765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:17.412267 13765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "f74c4dd494fb4feca3bde25c6291e68c"
format_stamp: "Formatted at 2026-09-28 13:08:17 on dist-test-slave-p1lc"
server_key: "912eacc892d623c4544c8b484e6b8d02"
server_key_iv: "6ba58cc390c940e3a7519cacaf7923cb"
server_key_version: "encryptionkey@0"
I20260928 13:08:17.412354 13765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:17.428419 13765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:17.428791 13765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:17.428982 13765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:17.433202 13765 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:44431
I20260928 13:08:17.433290 13831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:44431 every 8 connection(s)
I20260928 13:08:17.433658 13765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:17.434717 13832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:17.437356 13832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c: Bootstrap starting.
I20260928 13:08:17.438030 13832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:17.438329 13832 log.cc:824] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:17.439131 13832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c: No bootstrap required, opened a new log
I20260928 13:08:17.440860 13832 raft_consensus.cc:348] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74c4dd494fb4feca3bde25c6291e68c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44431 } }
I20260928 13:08:17.441031 13832 raft_consensus.cc:374] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:17.441056 13832 raft_consensus.cc:729] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74c4dd494fb4feca3bde25c6291e68c, State: Initialized, Role: FOLLOWER
I20260928 13:08:17.441215 13832 consensus_queue.cc:260] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74c4dd494fb4feca3bde25c6291e68c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44431 } }
I20260928 13:08:17.441280 13832 raft_consensus.cc:388] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:17.441303 13832 raft_consensus.cc:482] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:17.441371 13832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:17.441980 13832 raft_consensus.cc:504] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74c4dd494fb4feca3bde25c6291e68c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44431 } }
I20260928 13:08:17.442072 13832 leader_election.cc:302] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f74c4dd494fb4feca3bde25c6291e68c; no voters:
I20260928 13:08:17.442414 13832 leader_election.cc:288] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:17.442456 13835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:17.442562 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 13765
I20260928 13:08:17.442652 13835 raft_consensus.cc:686] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [term 1 LEADER]: Becoming Leader. State: Replica: f74c4dd494fb4feca3bde25c6291e68c, State: Running, Role: LEADER
I20260928 13:08:17.442665 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:17.442735 13832 sys_catalog.cc:564] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:17.442790 13835 consensus_queue.cc:237] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74c4dd494fb4feca3bde25c6291e68c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44431 } }
I20260928 13:08:17.442867 8546 external_mini_cluster.cc:1428] Setting key bb0486e2b8fc09ee7e66a1626441a728
I20260928 13:08:17.443158 13836 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f74c4dd494fb4feca3bde25c6291e68c. Latest consensus state: current_term: 1 leader_uuid: "f74c4dd494fb4feca3bde25c6291e68c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74c4dd494fb4feca3bde25c6291e68c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44431 } } }
I20260928 13:08:17.443241 13836 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:17.443593 13837 sys_catalog.cc:455] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f74c4dd494fb4feca3bde25c6291e68c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f74c4dd494fb4feca3bde25c6291e68c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44431 } } }
I20260928 13:08:17.443722 13837 sys_catalog.cc:458] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:17.444072 13843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:17.444631 13843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:17.446293 13843 catalog_manager.cc:1347] Generated new cluster ID: 0dc89784ff9b416e94b462a6b265bd2b
I20260928 13:08:17.446345 13843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:08:17.450146 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:17.443391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.8.88.190:44431)
0928 13:08:17.443968 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:17.443978 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:17.444073 (+ 95us) server_negotiation.cc:407] Connection header received
0928 13:08:17.444200 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:17.444208 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:17.444348 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:17.444537 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:17.444790 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.446369 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.447660 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.448385 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.448536 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:17.449351 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:17.449355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:17.449356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:17.449601 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:17.449646 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:17.449665 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:17.449741 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:17.449820 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 13:08:17.449907 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":90,"threads_started":1}
I20260928 13:08:17.451143 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:08:17.481761 13843 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:17.482347 13843 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:17.487862 13843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f74c4dd494fb4feca3bde25c6291e68c: Generated new TSK 0
I20260928 13:08:17.488101 13843 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 13:08:17.512820 13862 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 13:08:17.512943 13862 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 13:08:17.513309 13846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:17.497920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.8.88.190:44431)
0928 13:08:17.497983 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:17.497986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:17.498112 (+ 126us) server_negotiation.cc:407] Connection header received
0928 13:08:17.498216 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:17.498217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:17.498230 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:17.498321 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0928 13:08:17.498477 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.498989 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.499918 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.500233 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.500319 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:17.500519 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 13:08:17.513098 (+ 12579us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 13:08:17.513255 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 13:08:17.513386 13846 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 13:08:17.513506 13861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:08:17.497860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:44431 (local address 127.0.0.1:58990)
0928 13:08:17.498041 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:17.498051 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:17.498131 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:17.498320 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:17.498320 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:17.498323 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 13:08:17.498390 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:17.498393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.499130 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.499131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:17.499823 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:17.499826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.500324 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.500326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:17.500392 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:17.500397 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 13:08:17.513246 (+ 12849us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 13:08:17.513404 (+ 158us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.88.190:44431: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260928 13:08:17.513796 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 13765
2026-09-28T13:08:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (254 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 13:08:17.552675 8546 mini_oidc.cc:152] Starting JWKS server
I20260928 13:08:17.553486 8546 webserver.cc:460] Webserver started at https://127.0.0.1:33865/ using document root <none> and password file <none>
I20260928 13:08:17.553536 8546 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 13:08:17.553728 8546 webserver.cc:460] Webserver started at http://127.0.0.1:33267/ using document root <none> and password file <none>
2026-09-28T13:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:17Z Disabled control of system clock
I20260928 13:08:17.563221 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:37681
--webserver_interface=127.8.88.190
--webserver_port=0
--builtin_ntp_servers=127.8.88.148:45949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:37681
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33865/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 13:08:17.650590 13868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:17.650820 13868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:17.650846 13868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:17.652045 13868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:17.652086 13868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:17.652098 13868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:17.652112 13868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:17.652123 13868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 13:08:17.652135 13868 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 13:08:17.652146 13868 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33865/jwks.json
I20260928 13:08:17.653601 13868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:45949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:37681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:37681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33865/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:17.653820 13868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:17.654073 13868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:17.659200 13876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:17.659780 13868 server_base.cc:1034] running on GCE node
W20260928 13:08:17.659847 13874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:17.659848 13873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:17.660250 13868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:17.660550 13868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:17.661706 13868 hybrid_clock.cc:648] HybridClock initialized: now 1790600897661680 us; error 49 us; skew 500 ppm
I20260928 13:08:17.663090 13868 webserver.cc:460] Webserver started at http://127.8.88.190:32943/ using document root <none> and password file <none>
I20260928 13:08:17.663307 13868 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:17.663349 13868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:17.663465 13868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:17.664585 13868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "1e97261becdb40b99f442da70af95ee5"
format_stamp: "Formatted at 2026-09-28 13:08:17 on dist-test-slave-p1lc"
server_key: "22216e3f3fe6f1f947699a840dd69dd0"
server_key_iv: "2e96e19d4ecbdb1f5a5a9a0c0e97b86e"
server_key_version: "encryptionkey@0"
I20260928 13:08:17.664983 13868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "1e97261becdb40b99f442da70af95ee5"
format_stamp: "Formatted at 2026-09-28 13:08:17 on dist-test-slave-p1lc"
server_key: "22216e3f3fe6f1f947699a840dd69dd0"
server_key_iv: "2e96e19d4ecbdb1f5a5a9a0c0e97b86e"
server_key_version: "encryptionkey@0"
I20260928 13:08:17.666270 13868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:17.666982 13882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:17.667169 13868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:17.667256 13868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "1e97261becdb40b99f442da70af95ee5"
format_stamp: "Formatted at 2026-09-28 13:08:17 on dist-test-slave-p1lc"
server_key: "22216e3f3fe6f1f947699a840dd69dd0"
server_key_iv: "2e96e19d4ecbdb1f5a5a9a0c0e97b86e"
server_key_version: "encryptionkey@0"
I20260928 13:08:17.667336 13868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:17.688165 13868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:17.688477 13868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:17.688587 13868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:17.693320 13868 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:37681
I20260928 13:08:17.693540 13934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:37681 every 8 connection(s)
I20260928 13:08:17.693789 13868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:17.694697 13935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:17.698287 13935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5: Bootstrap starting.
I20260928 13:08:17.698850 13935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:17.699106 13935 log.cc:824] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:17.699283 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 13868
I20260928 13:08:17.699515 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:17.699713 8546 external_mini_cluster.cc:1428] Setting key 080b441515ccdbd36d43b0ae27fcb7fa
I20260928 13:08:17.700018 13935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5: No bootstrap required, opened a new log
I20260928 13:08:17.703146 13935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e97261becdb40b99f442da70af95ee5" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37681 } }
I20260928 13:08:17.703336 13935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:17.703367 13935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e97261becdb40b99f442da70af95ee5, State: Initialized, Role: FOLLOWER
I20260928 13:08:17.703527 13935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e97261becdb40b99f442da70af95ee5" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37681 } }
I20260928 13:08:17.703622 13935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:17.703696 13935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:17.703750 13935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:17.706521 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:17.700296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.8.88.190:37681)
0928 13:08:17.700537 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:17.700546 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:17.700590 (+ 44us) server_negotiation.cc:407] Connection header received
0928 13:08:17.701589 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:17.701599 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:17.701687 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:17.701860 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:17.702112 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.703570 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.704547 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.705087 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.705185 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:17.705834 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:17.705837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:17.705838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:17.706039 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:17.706074 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:17.706085 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:17.706144 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:17.706211 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 13:08:17.706308 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":98,"threads_started":1}
I20260928 13:08:17.707623 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:08:17.778561 13935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e97261becdb40b99f442da70af95ee5" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37681 } }
I20260928 13:08:17.778733 13935 leader_election.cc:302] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e97261becdb40b99f442da70af95ee5; no voters:
I20260928 13:08:17.779029 13935 leader_election.cc:288] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:17.779232 13940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:17.779372 13935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:17.779501 13940 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [term 1 LEADER]: Becoming Leader. State: Replica: 1e97261becdb40b99f442da70af95ee5, State: Running, Role: LEADER
I20260928 13:08:17.779659 13940 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e97261becdb40b99f442da70af95ee5" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37681 } }
I20260928 13:08:17.780122 13941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e97261becdb40b99f442da70af95ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e97261becdb40b99f442da70af95ee5" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37681 } } }
I20260928 13:08:17.780237 13941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:17.780383 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e97261becdb40b99f442da70af95ee5. Latest consensus state: current_term: 1 leader_uuid: "1e97261becdb40b99f442da70af95ee5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e97261becdb40b99f442da70af95ee5" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37681 } } }
I20260928 13:08:17.780457 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:17.780684 13949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:17.781328 13949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:17.783324 13949 catalog_manager.cc:1347] Generated new cluster ID: 7924a2d5c1844a679b57442168d5b16f
I20260928 13:08:17.783389 13949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:08:17.832639 13949 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:17.833446 13949 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:17.841840 13949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e97261becdb40b99f442da70af95ee5: Generated new TSK 0
I20260928 13:08:17.842109 13949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:08:17.857439 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:17.853348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56012 (local address 127.8.88.190:37681)
0928 13:08:17.853398 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:17.853401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:17.853534 (+ 133us) server_negotiation.cc:407] Connection header received
0928 13:08:17.853705 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:17.853706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:17.853719 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:17.853796 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0928 13:08:17.854013 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.854506 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.855749 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.855942 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.856005 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:17.856218 (+ 213us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 13:08:17.857201 (+ 983us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 13:08:17.857359 (+ 158us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56012: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 13:08:17.857510 13939 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56012: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 13:08:17.857522 13964 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:08:17.853240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:37681 (local address 127.0.0.1:56012)
0928 13:08:17.853437 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:17.853451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:17.853540 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:17.853825 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:17.853826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:17.853829 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 13:08:17.853895 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:17.853898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.854630 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.854633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:17.855657 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:17.855661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:17.856041 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:17.856042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:17.856109 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:17.856116 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 13:08:17.857310 (+ 1194us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 13:08:17.857405 (+ 95us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.88.190:37681: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":110,"threads_started":1}
I20260928 13:08:17.858161 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 13868
E20260928 13:08:17.869336 13965 webserver.cc:533] Webserver: sslize failed:
E20260928 13:08:17.869429 13965 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T13:08:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (344 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T13:08:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:17Z Disabled control of system clock
I20260928 13:08:17.905686 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:32783
--webserver_interface=127.8.88.190
--webserver_port=0
--builtin_ntp_servers=127.8.88.148:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:32783
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/testchainca.cert with env {}
W20260928 13:08:17.994385 13969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:17.994648 13969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:17.994676 13969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:17.996042 13969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:17.996105 13969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:17.996119 13969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:17.996132 13969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:17.996145 13969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 13:08:17.996160 13969 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 13:08:17.996171 13969 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 13:08:17.998198 13969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:32783
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:32783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:17.998497 13969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:17.998764 13969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:18.004359 13975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:18.004359 13974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:18.004385 13977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:18.004974 13969 server_base.cc:1034] running on GCE node
I20260928 13:08:18.005297 13969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:18.005635 13969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:18.006795 13969 hybrid_clock.cc:648] HybridClock initialized: now 1790600898006776 us; error 43 us; skew 500 ppm
I20260928 13:08:18.008225 13969 webserver.cc:460] Webserver started at http://127.8.88.190:43661/ using document root <none> and password file <none>
I20260928 13:08:18.008703 13969 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:18.008759 13969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:18.008945 13969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:18.010238 13969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "1cd425bc0017409e8c83d3ccff63a0cc"
format_stamp: "Formatted at 2026-09-28 13:08:18 on dist-test-slave-p1lc"
server_key: "5ac9606ae83766ab492e329805a5bd12"
server_key_iv: "930dfd650b5415f41e07aaac2a0e3f51"
server_key_version: "encryptionkey@0"
I20260928 13:08:18.010654 13969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "1cd425bc0017409e8c83d3ccff63a0cc"
format_stamp: "Formatted at 2026-09-28 13:08:18 on dist-test-slave-p1lc"
server_key: "5ac9606ae83766ab492e329805a5bd12"
server_key_iv: "930dfd650b5415f41e07aaac2a0e3f51"
server_key_version: "encryptionkey@0"
I20260928 13:08:18.012226 13969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:08:18.013154 13983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:18.013374 13969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:18.013475 13969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "1cd425bc0017409e8c83d3ccff63a0cc"
format_stamp: "Formatted at 2026-09-28 13:08:18 on dist-test-slave-p1lc"
server_key: "5ac9606ae83766ab492e329805a5bd12"
server_key_iv: "930dfd650b5415f41e07aaac2a0e3f51"
server_key_version: "encryptionkey@0"
I20260928 13:08:18.013562 13969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:18.030289 13969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:18.030772 13969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:18.030920 13969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:18.035503 13969 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:32783
I20260928 13:08:18.035609 14035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:32783 every 8 connection(s)
I20260928 13:08:18.035986 13969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:18.037360 14036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:18.040010 14036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc: Bootstrap starting.
I20260928 13:08:18.040676 14036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:18.041031 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 13969
I20260928 13:08:18.041059 14036 log.cc:824] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:18.041112 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:18.041337 8546 external_mini_cluster.cc:1428] Setting key 70e34a40c21d4c81630418b22f8f9738
I20260928 13:08:18.042264 14036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc: No bootstrap required, opened a new log
I20260928 13:08:18.045230 14036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd425bc0017409e8c83d3ccff63a0cc" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 32783 } }
I20260928 13:08:18.045397 14036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:18.045423 14036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd425bc0017409e8c83d3ccff63a0cc, State: Initialized, Role: FOLLOWER
I20260928 13:08:18.045568 14036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd425bc0017409e8c83d3ccff63a0cc" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 32783 } }
I20260928 13:08:18.045662 14036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:18.045687 14036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:18.045761 14036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:18.048259 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:18.041932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47724 (local address 127.8.88.190:32783)
0928 13:08:18.042285 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:18.042296 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:18.042332 (+ 36us) server_negotiation.cc:407] Connection header received
0928 13:08:18.042920 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:18.042930 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:18.043016 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:18.043238 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:18.043413 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:18.044945 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:18.046098 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:18.046774 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:18.046882 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:18.047620 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:18.047624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:18.047625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:08:18.047806 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:18.047841 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:08:18.047853 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:18.047907 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:18.048009 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 13:08:18.048103 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":94,"threads_started":1}
I20260928 13:08:18.049341 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:08:18.122992 14036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd425bc0017409e8c83d3ccff63a0cc" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 32783 } }
I20260928 13:08:18.123179 14036 leader_election.cc:302] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cd425bc0017409e8c83d3ccff63a0cc; no voters:
I20260928 13:08:18.123510 14036 leader_election.cc:288] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:18.123605 14041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:18.123811 14041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [term 1 LEADER]: Becoming Leader. State: Replica: 1cd425bc0017409e8c83d3ccff63a0cc, State: Running, Role: LEADER
I20260928 13:08:18.123879 14036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:18.123934 14041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd425bc0017409e8c83d3ccff63a0cc" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 32783 } }
I20260928 13:08:18.124472 14043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cd425bc0017409e8c83d3ccff63a0cc. Latest consensus state: current_term: 1 leader_uuid: "1cd425bc0017409e8c83d3ccff63a0cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd425bc0017409e8c83d3ccff63a0cc" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 32783 } } }
I20260928 13:08:18.124513 14042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cd425bc0017409e8c83d3ccff63a0cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd425bc0017409e8c83d3ccff63a0cc" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 32783 } } }
I20260928 13:08:18.124635 14043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:18.124660 14042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:18.125048 14049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:18.125643 14049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:18.127745 14049 catalog_manager.cc:1347] Generated new cluster ID: 3f1c1fc0eb4646c29da1b0c9de0b4ed6
I20260928 13:08:18.127830 14049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:08:18.140213 14049 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:18.140795 14049 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:18.144958 14049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cd425bc0017409e8c83d3ccff63a0cc: Generated new TSK 0
I20260928 13:08:18.145218 14049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:08:18.211333 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:18.158546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47732 (local address 127.8.88.190:32783)
0928 13:08:18.158616 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:18.158618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:18.158779 (+ 161us) server_negotiation.cc:407] Connection header received
0928 13:08:18.158837 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:18.158838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:18.158853 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:18.158948 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0928 13:08:18.159326 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:18.160229 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:18.161186 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:18.161525 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:18.161605 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:18.161806 (+ 201us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 13:08:18.211013 (+ 49207us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 13:08:18.211248 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47732: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 13:08:18.211416 14065 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:08:18.158447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:32783 (local address 127.0.0.1:47732)
0928 13:08:18.158708 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:18.158718 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:18.158775 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:18.159177 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:18.159178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:18.159180 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 13:08:18.159241 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:18.159244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:18.160345 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:18.160346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:18.161100 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:18.161103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:18.161626 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:18.161627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:18.161680 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:18.161687 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 13:08:18.211179 (+ 49492us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 13:08:18.211303 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.88.190:32783: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":200,"threads_started":1}
W20260928 13:08:18.211431 14040 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47732: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260928 13:08:18.211817 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 13969
2026-09-28T13:08:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (332 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (98 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (97 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:08:18 dist-test-slave-p1lc krb5kdc[14077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:08:18 dist-test-slave-p1lc krb5kdc[14077](info): set up 2 sockets
Sep 28 13:08:18 dist-test-slave-p1lc krb5kdc[14077](info): commencing operation
krb5kdc: starting...
W20260928 13:08:19.461457 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:08:19 dist-test-slave-p1lc krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:08:19.593039 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:44729
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:44729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:19.678274 14090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:19.678553 14090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:19.678597 14090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:19.679885 14090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:19.679970 14090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:19.679994 14090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:19.680017 14090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:19.680045 14090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:19.681725 14090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:44729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:44729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:19.682052 14090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:19.682307 14090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:19.687562 14095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:19.687575 14096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:19.687608 14098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:19.687841 14090 server_base.cc:1034] running on GCE node
I20260928 13:08:19.688196 14090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:19.688575 14090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:19.689730 14090 hybrid_clock.cc:648] HybridClock initialized: now 1790600899689710 us; error 43 us; skew 500 ppm
Sep 28 13:08:19 dist-test-slave-p1lc krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600899, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:19.692615 14090 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:08:19.693219 14090 webserver.cc:460] Webserver started at http://127.8.88.190:44951/ using document root <none> and password file <none>
I20260928 13:08:19.693490 14090 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:19.693563 14090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:19.693714 14090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:19.694840 14090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "83065b80db7744bf81b0a0e98cd67d30"
format_stamp: "Formatted at 2026-09-28 13:08:19 on dist-test-slave-p1lc"
server_key: "0dc531411d8c4db7ab6b9e5c99a8b206"
server_key_iv: "0b76e1af2bcdc40faaa985b9cc8337c5"
server_key_version: "encryptionkey@0"
I20260928 13:08:19.695222 14090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "83065b80db7744bf81b0a0e98cd67d30"
format_stamp: "Formatted at 2026-09-28 13:08:19 on dist-test-slave-p1lc"
server_key: "0dc531411d8c4db7ab6b9e5c99a8b206"
server_key_iv: "0b76e1af2bcdc40faaa985b9cc8337c5"
server_key_version: "encryptionkey@0"
I20260928 13:08:19.696709 14090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:08:19.697616 14105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:19.697813 14090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:19.697919 14090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "83065b80db7744bf81b0a0e98cd67d30"
format_stamp: "Formatted at 2026-09-28 13:08:19 on dist-test-slave-p1lc"
server_key: "0dc531411d8c4db7ab6b9e5c99a8b206"
server_key_iv: "0b76e1af2bcdc40faaa985b9cc8337c5"
server_key_version: "encryptionkey@0"
I20260928 13:08:19.698005 14090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:19.718160 14090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:19.720896 14090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:19.721115 14090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:19.725383 14090 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:44729
I20260928 13:08:19.725550 14157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:44729 every 8 connection(s)
I20260928 13:08:19.725872 14090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:19.726866 14158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:19.728756 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 14090
I20260928 13:08:19.728840 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:19.729115 8546 external_mini_cluster.cc:1428] Setting key 27ef1b6b37a6679d8141b476b382982c
I20260928 13:08:19.729562 14158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30: Bootstrap starting.
I20260928 13:08:19.730248 14158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:19.730583 14158 log.cc:824] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:19.731447 14158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30: No bootstrap required, opened a new log
I20260928 13:08:19.733706 14158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83065b80db7744bf81b0a0e98cd67d30" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44729 } }
I20260928 13:08:19.733880 14158 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:19.733906 14158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83065b80db7744bf81b0a0e98cd67d30, State: Initialized, Role: FOLLOWER
I20260928 13:08:19.734051 14158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83065b80db7744bf81b0a0e98cd67d30" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44729 } }
I20260928 13:08:19.734112 14158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:19.734134 14158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:19.734205 14158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:19.734973 14158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83065b80db7744bf81b0a0e98cd67d30" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44729 } }
I20260928 13:08:19.735112 14158 leader_election.cc:302] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83065b80db7744bf81b0a0e98cd67d30; no voters:
I20260928 13:08:19.735291 14158 leader_election.cc:288] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:19.735360 14163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:19.735584 14163 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [term 1 LEADER]: Becoming Leader. State: Replica: 83065b80db7744bf81b0a0e98cd67d30, State: Running, Role: LEADER
I20260928 13:08:19.735589 14158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:19.735699 14163 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83065b80db7744bf81b0a0e98cd67d30" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44729 } }
I20260928 13:08:19.736079 14164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83065b80db7744bf81b0a0e98cd67d30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83065b80db7744bf81b0a0e98cd67d30" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44729 } } }
I20260928 13:08:19.736214 14164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:19.736068 14163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83065b80db7744bf81b0a0e98cd67d30. Latest consensus state: current_term: 1 leader_uuid: "83065b80db7744bf81b0a0e98cd67d30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83065b80db7744bf81b0a0e98cd67d30" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44729 } } }
I20260928 13:08:19.736404 14163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:19.736433 14171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 13:08:19 dist-test-slave-p1lc krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:19.737043 14171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:19.741585 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:19.729777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.8.88.190:44729)
0928 13:08:19.729975 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:19.729985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:19.730025 (+ 40us) server_negotiation.cc:407] Connection header received
0928 13:08:19.730554 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:19.730562 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:19.730615 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:19.730779 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:19.731576 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:19.732516 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:19.733745 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:19.734421 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:19.734532 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:19.737762 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:19.737766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:19.737767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:19.737778 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:19.739772 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:19.740229 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:19.740232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:19.740232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:19.740320 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:19.740568 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:19.740569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:19.740569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:19.740972 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:19.741150 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:19.741223 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 13:08:19.741365 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":112,"threads_started":1}
I20260928 13:08:19.741676 14171 catalog_manager.cc:1347] Generated new cluster ID: 00c0778d41384a07b13b235d642e079f
I20260928 13:08:19.741837 14171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:08:19.767777 14171 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:19.768487 14171 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:19.773141 14171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83065b80db7744bf81b0a0e98cd67d30: Generated new TSK 0
I20260928 13:08:19.773358 14171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:08:19.807559 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44729
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:19.905346 14184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:19.905572 14184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:19.905599 14184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:19.906781 14184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:19.906827 14184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:19.906857 14184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:08:19.908355 14184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:19.908603 14184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:19.908792 14184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:19.914117 14192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:19.914109 14190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:19.914201 14184 server_base.cc:1034] running on GCE node
W20260928 13:08:19.914100 14189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:19.914638 14184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:19.914860 14184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:19.916071 14184 hybrid_clock.cc:648] HybridClock initialized: now 1790600899916043 us; error 41 us; skew 500 ppm
Sep 28 13:08:19 dist-test-slave-p1lc krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600899, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:19.919533 14184 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:08:19.920161 14184 webserver.cc:460] Webserver started at http://127.8.88.129:38731/ using document root <none> and password file <none>
I20260928 13:08:19.920411 14184 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:19.920460 14184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:19.920611 14184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:19.921813 14184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "5b9758f97c5a4d1eb538567b22598edd"
format_stamp: "Formatted at 2026-09-28 13:08:19 on dist-test-slave-p1lc"
server_key: "c040d9b8dcf1eba7b287c96253292177"
server_key_iv: "7d1f3172e12bec13bf2f76113d73255d"
server_key_version: "encryptionkey@0"
I20260928 13:08:19.922195 14184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "5b9758f97c5a4d1eb538567b22598edd"
format_stamp: "Formatted at 2026-09-28 13:08:19 on dist-test-slave-p1lc"
server_key: "c040d9b8dcf1eba7b287c96253292177"
server_key_iv: "7d1f3172e12bec13bf2f76113d73255d"
server_key_version: "encryptionkey@0"
I20260928 13:08:19.923813 14184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:08:19.924821 14199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:19.925107 14184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:19.925199 14184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "5b9758f97c5a4d1eb538567b22598edd"
format_stamp: "Formatted at 2026-09-28 13:08:19 on dist-test-slave-p1lc"
server_key: "c040d9b8dcf1eba7b287c96253292177"
server_key_iv: "7d1f3172e12bec13bf2f76113d73255d"
server_key_version: "encryptionkey@0"
I20260928 13:08:19.925258 14184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:19.938313 14184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:19.940807 14184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:19.941035 14184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:19.941263 14184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:19.941620 14184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:19.941679 14184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:19.941727 14184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:19.941754 14184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:19.947564 14184 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:33705
I20260928 13:08:19.947685 14312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:33705 every 8 connection(s)
I20260928 13:08:19.947986 14184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:08:19.953678 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 14184
I20260928 13:08:19.953774 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:08:19.953975 8546 external_mini_cluster.cc:1428] Setting key ea6af392f6dbc18d98ade34879030b5d
Sep 28 13:08:19 dist-test-slave-p1lc krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600899, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:19.959324 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:19.948824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:40055 (local address 127.8.88.190:44729)
0928 13:08:19.948927 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:19.948929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:19.949170 (+ 241us) server_negotiation.cc:407] Connection header received
0928 13:08:19.950197 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:19.950198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:19.950210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:19.950273 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:19.950892 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:19.951420 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:19.952532 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:19.952731 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:19.952798 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:19.955819 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:19.955820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:19.955820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:19.955831 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:19.957422 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:19.957874 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:19.957876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:19.957876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:19.957941 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:19.958303 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:19.958304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:19.958304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:19.958575 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:19.958686 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:19.959054 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 13:08:19.959208 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 13:08:19.959527 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:19.948790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:44729 (local address 127.8.88.129:40055)
0928 13:08:19.948993 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:19.949037 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:19.950013 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:19.950332 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:19.950335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:19.950610 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:19.950806 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:19.950810 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:19.951526 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:19.951527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:19.952434 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:19.952436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:19.952818 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:19.952818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:19.952992 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:19.953634 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:19.953643 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:19.955701 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:19.957540 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:19.957548 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:19.957551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:19.957790 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:19.958049 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:19.958051 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:19.958051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:19.958226 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:19.958678 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:19.958686 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:19.958878 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 13:08:19.959017 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 13:08:19.959221 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":128,"threads_started":1}
I20260928 13:08:19.960258 14313 heartbeater.cc:347] Connected to a master server at 127.8.88.190:44729
I20260928 13:08:19.960397 14313 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:19.960683 14313 heartbeater.cc:511] Master 127.8.88.190:44729 requested a full tablet report, sending...
I20260928 13:08:19.961614 14122 ts_manager.cc:194] Registered new tserver with Master: 5b9758f97c5a4d1eb538567b22598edd (127.8.88.129:33705)
I20260928 13:08:19.962321 14122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:40055
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:08:19.984017 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44729
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:20.069451 14318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:20.069731 14318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:20.069777 14318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:20.071048 14318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:20.071130 14318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:20.071178 14318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:08:20.073266 14318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:20.073673 14318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:20.073941 14318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:20.079646 14326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:20.079660 14324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:20.079646 14323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:20.080014 14318 server_base.cc:1034] running on GCE node
I20260928 13:08:20.080243 14318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:20.080595 14318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:20.081786 14318 hybrid_clock.cc:648] HybridClock initialized: now 1790600900081768 us; error 63 us; skew 500 ppm
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:20.084774 14318 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:08:20.085350 14318 webserver.cc:460] Webserver started at http://127.8.88.130:44241/ using document root <none> and password file <none>
I20260928 13:08:20.085665 14318 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:20.085747 14318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:20.085911 14318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:20.087021 14318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "f8ed39ead9eb4034ae6de3d745470099"
format_stamp: "Formatted at 2026-09-28 13:08:20 on dist-test-slave-p1lc"
server_key: "135a9f41611b97da8ffc887875151a7c"
server_key_iv: "cd86e58b21cb863478526167a93ce864"
server_key_version: "encryptionkey@0"
I20260928 13:08:20.087412 14318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "f8ed39ead9eb4034ae6de3d745470099"
format_stamp: "Formatted at 2026-09-28 13:08:20 on dist-test-slave-p1lc"
server_key: "135a9f41611b97da8ffc887875151a7c"
server_key_iv: "cd86e58b21cb863478526167a93ce864"
server_key_version: "encryptionkey@0"
I20260928 13:08:20.089007 14318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 13:08:20.090317 14333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:20.090615 14318 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:20.090749 14318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "f8ed39ead9eb4034ae6de3d745470099"
format_stamp: "Formatted at 2026-09-28 13:08:20 on dist-test-slave-p1lc"
server_key: "135a9f41611b97da8ffc887875151a7c"
server_key_iv: "cd86e58b21cb863478526167a93ce864"
server_key_version: "encryptionkey@0"
I20260928 13:08:20.090857 14318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:20.109207 14318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:20.111919 14318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:20.112071 14318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:20.112413 14318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:20.112805 14318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:20.112848 14318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:20.112929 14318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:20.112960 14318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:20.120216 14318 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:33319
I20260928 13:08:20.120256 14446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:33319 every 8 connection(s)
I20260928 13:08:20.120672 14318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:20.129900 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 14318
I20260928 13:08:20.130028 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:08:20.130256 8546 external_mini_cluster.cc:1428] Setting key 3970b56b4b31bdf0a5d6a2525f3f3056
I20260928 13:08:20.132002 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:20.121591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:54037 (local address 127.8.88.190:44729)
0928 13:08:20.121665 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:20.121668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:20.121995 (+ 327us) server_negotiation.cc:407] Connection header received
0928 13:08:20.123001 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:20.123002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:20.123014 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:20.123101 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:20.123788 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.124326 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.125587 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.125775 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.125829 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:20.128344 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:20.128345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:20.128345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:20.128353 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:20.130388 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:20.130901 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:20.130903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:20.130903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:20.130969 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:20.131282 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:20.131283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:20.131283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:20.131406 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:20.131483 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:20.131812 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 13:08:20.131903 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 13:08:20.132292 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:20.121642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:44729 (local address 127.8.88.130:54037)
0928 13:08:20.121858 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:20.121900 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:20.122888 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:20.123178 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:20.123182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:20.123472 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:20.123699 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:20.123705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.124422 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.124423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:20.125484 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:20.125487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.125847 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.125848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:20.125951 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:20.126551 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:20.126561 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:20.128242 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:20.130553 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:20.130563 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:20.130565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:20.130829 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:20.131067 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:20.131068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:20.131069 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:20.131199 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:20.131474 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:20.131482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:20.131668 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 13:08:20.131791 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 13:08:20.131970 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260928 13:08:20.132987 14447 heartbeater.cc:347] Connected to a master server at 127.8.88.190:44729
I20260928 13:08:20.133108 14447 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:20.133298 14447 heartbeater.cc:511] Master 127.8.88.190:44729 requested a full tablet report, sending...
I20260928 13:08:20.133870 14122 ts_manager.cc:194] Registered new tserver with Master: f8ed39ead9eb4034ae6de3d745470099 (127.8.88.130:33319)
I20260928 13:08:20.134289 14122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:54037
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:08:20.161217 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44729
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:41297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:20.249166 14452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:20.249396 14452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:20.249431 14452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:20.250689 14452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:20.250738 14452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:20.250769 14452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:08:20.252363 14452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:20.252646 14452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:20.252940 14452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:20.258589 14460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:20.258581 14457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:20.258579 14458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:20.258579 14452 server_base.cc:1034] running on GCE node
I20260928 13:08:20.259091 14452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:20.259357 14452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:20.260597 14452 hybrid_clock.cc:648] HybridClock initialized: now 1790600900260582 us; error 30 us; skew 500 ppm
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:20.263422 14452 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:08:20.264005 14452 webserver.cc:460] Webserver started at http://127.8.88.131:33057/ using document root <none> and password file <none>
I20260928 13:08:20.264251 14452 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:20.264318 14452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:20.264468 14452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:20.265645 14452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "33acda1ffd68441ba8ba64070500d15d"
format_stamp: "Formatted at 2026-09-28 13:08:20 on dist-test-slave-p1lc"
server_key: "48f4b0a0e00dcd9b42c07ea7012ab314"
server_key_iv: "fd0e642a449974b2681703f4353eedbf"
server_key_version: "encryptionkey@0"
I20260928 13:08:20.266026 14452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "33acda1ffd68441ba8ba64070500d15d"
format_stamp: "Formatted at 2026-09-28 13:08:20 on dist-test-slave-p1lc"
server_key: "48f4b0a0e00dcd9b42c07ea7012ab314"
server_key_iv: "fd0e642a449974b2681703f4353eedbf"
server_key_version: "encryptionkey@0"
I20260928 13:08:20.267551 14452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:08:20.268447 14467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:20.268760 14452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:20.268908 14452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "33acda1ffd68441ba8ba64070500d15d"
format_stamp: "Formatted at 2026-09-28 13:08:20 on dist-test-slave-p1lc"
server_key: "48f4b0a0e00dcd9b42c07ea7012ab314"
server_key_iv: "fd0e642a449974b2681703f4353eedbf"
server_key_version: "encryptionkey@0"
I20260928 13:08:20.268996 14452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:20.308561 14452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:20.311040 14452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:20.311213 14452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:20.311468 14452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:20.311802 14452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:20.311853 14452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:20.311889 14452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:20.311931 14452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:20.318019 14452 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:43343
I20260928 13:08:20.318081 14580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:43343 every 8 connection(s)
I20260928 13:08:20.318429 14452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:20.326943 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 14452
I20260928 13:08:20.327098 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:08:20.327296 8546 external_mini_cluster.cc:1428] Setting key 62de9a8aca27e7b168ea548d2b00993e
I20260928 13:08:20.329222 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:20.319246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:60417 (local address 127.8.88.190:44729)
0928 13:08:20.319303 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:20.319305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:20.319481 (+ 176us) server_negotiation.cc:407] Connection header received
0928 13:08:20.320484 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:20.320486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:20.320496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:20.320579 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:20.321342 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.321862 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.322979 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.323176 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.323240 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:20.325783 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:20.325784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:20.325785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:20.325792 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:20.327585 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:20.328155 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:20.328156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:20.328156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:20.328223 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:20.328490 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:20.328491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:20.328491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:20.328652 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:20.328707 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:20.329090 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 13:08:20.329158 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 13:08:20.329464 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:20.319201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:44729 (local address 127.8.88.131:60417)
0928 13:08:20.319356 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:20.319400 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:20.320333 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:20.320656 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:20.320660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:20.321019 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:20.321236 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:20.321242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.321965 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.321965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:20.322867 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:20.322869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.323277 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.323279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:20.323403 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:20.324002 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:20.324015 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:20.325669 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:20.327711 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:20.327717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:20.327719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:20.328090 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:20.328287 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:20.328287 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:20.328288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:20.328435 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:20.328711 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:20.328716 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:20.328952 (+ 236us) client_negotiation.cc:769] Sending connection context
0928 13:08:20.329061 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 13:08:20.329194 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260928 13:08:20.329977 14581 heartbeater.cc:347] Connected to a master server at 127.8.88.190:44729
I20260928 13:08:20.330109 14581 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:20.330296 14581 heartbeater.cc:511] Master 127.8.88.190:44729 requested a full tablet report, sending...
I20260928 13:08:20.330669 14122 ts_manager.cc:194] Registered new tserver with Master: 33acda1ffd68441ba8ba64070500d15d (127.8.88.131:43343)
I20260928 13:08:20.331089 14122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:60417
I20260928 13:08:20.339623 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/test-admin.keytab.
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:20.358726 8546 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 13:08:20.360519 14588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:08:20.359644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.8.88.129:33705)
0928 13:08:20.359824 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:20.359828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:20.359838 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:08:20.359866 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:20.359868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:20.359883 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:20.359954 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:20.360360 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33670: BlockingRecv error: recv got EOF from 127.0.0.1:33670 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
W20260928 13:08:20.360529 8546 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/krb5cc' of type FILE
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:20.361593 8546 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:20.362478 8546 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.129@KRBTEST.COM
I20260928 13:08:20.370909 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:20.363002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.8.88.129:33705)
0928 13:08:20.363095 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:20.363097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:20.363183 (+ 86us) server_negotiation.cc:407] Connection header received
0928 13:08:20.363268 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:20.363269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:20.363285 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:20.363346 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:20.363771 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.364561 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.365393 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.365735 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.365815 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:20.367970 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:20.367981 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:20.367981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:20.367990 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:20.369722 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:20.370088 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:20.370092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:20.370093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:20.370148 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:20.370343 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:20.370344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:20.370345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:20.370487 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:20.370659 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:20.370704 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 13:08:20.370788 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 13:08:20.372861 14588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:08:20.372318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.8.88.129:33705)
0928 13:08:20.372352 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:20.372354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:20.372403 (+ 49us) server_negotiation.cc:407] Connection header received
0928 13:08:20.372460 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:20.372461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:20.372469 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:20.372522 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:20.372811 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33702: BlockingRecv error: recv got EOF from 127.0.0.1:33702 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 13:08:20.372965 8546 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/krb5cc' of type FILE
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:20.374043 8546 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:20.374883 8546 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.129@KRBTEST.COM
I20260928 13:08:20.382678 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:20.375436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.8.88.129:33705)
0928 13:08:20.375473 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:20.375475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:20.375514 (+ 39us) server_negotiation.cc:407] Connection header received
0928 13:08:20.375583 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:20.375583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:20.375591 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:20.375639 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:20.376016 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.376576 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.377390 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.377644 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.377701 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:20.379918 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:20.379920 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:20.379920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:20.379927 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:20.381388 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:20.381790 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:20.381791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:20.381792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:20.381848 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:20.382120 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:20.382121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:20.382121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:20.382218 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:20.382295 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:20.382442 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 13:08:20.382556 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 13:08:20.384470 14588 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:08:20.383863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.8.88.129:33705)
0928 13:08:20.383929 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:20.383931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:20.383968 (+ 37us) server_negotiation.cc:407] Connection header received
0928 13:08:20.384027 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:20.384028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:20.384036 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:20.384081 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:20.384407 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33724: BlockingRecv error: recv got EOF from 127.0.0.1:33724 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 13:08:20.384529 8546 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790600874459249-8546-0/krb5kdc/krb5cc' of type FILE
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:20.385670 8546 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:20.386521 8546 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.129@KRBTEST.COM
I20260928 13:08:20.394189 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:20.387167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33740 (local address 127.8.88.129:33705)
0928 13:08:20.387229 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:20.387231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:20.387240 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:08:20.387275 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:20.387276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:20.387286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:20.387361 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:20.387759 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.388256 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.389115 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:20.389311 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:20.389391 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:20.391455 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:20.391456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:20.391457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:20.391465 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:20.393022 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:20.393383 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:20.393385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:20.393386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:20.393441 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:20.393631 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:20.393632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:20.393632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:20.393745 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:20.393850 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:20.394017 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 13:08:20.394086 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 13:08:20.394938 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 14184
I20260928 13:08:20.399822 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 14318
I20260928 13:08:20.405051 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 14452
I20260928 13:08:20.410022 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 14090
2026-09-28T13:08:20Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2005 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14601](info): set up 2 sockets
Sep 28 13:08:20 dist-test-slave-p1lc krb5kdc[14601](info): commencing operation
krb5kdc: starting...
W20260928 13:08:21.497567 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:08:21 dist-test-slave-p1lc krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:08:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.190.keytab.
Entry for principal oryx/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.190.keytab.
I20260928 13:08:21.641096 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:43479
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.190.keytab
--principal=oryx/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:43479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:21.725598 14614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:21.725869 14614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:21.725912 14614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:21.727167 14614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:21.727246 14614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:21.727270 14614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:21.727293 14614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:21.727322 14614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:21.729005 14614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:43479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.190.keytab
--principal=oryx/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:43479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:21.729326 14614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:21.729573 14614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:21.734839 14622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:21.734845 14620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:21.734839 14619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:21.735098 14614 server_base.cc:1034] running on GCE node
I20260928 13:08:21.735348 14614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:21.735806 14614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:21.736958 14614 hybrid_clock.cc:648] HybridClock initialized: now 1790600901736930 us; error 48 us; skew 500 ppm
Sep 28 13:08:21 dist-test-slave-p1lc krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600901, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:21.745298 14614 init.cc:377] Logged in from keytab as oryx/127.8.88.190@KRBTEST.COM (short username oryx)
I20260928 13:08:21.745904 14614 webserver.cc:460] Webserver started at http://127.8.88.190:46717/ using document root <none> and password file <none>
I20260928 13:08:21.746174 14614 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:21.746248 14614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:21.746399 14614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:21.747517 14614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "c81d9bc77ee54b4c8d04746984c5ed6e"
format_stamp: "Formatted at 2026-09-28 13:08:21 on dist-test-slave-p1lc"
server_key: "c335e589df4e74deecc838aad563a434"
server_key_iv: "756d0dad81f72f11aeeb7fa8ea447ecc"
server_key_version: "encryptionkey@0"
I20260928 13:08:21.747903 14614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "c81d9bc77ee54b4c8d04746984c5ed6e"
format_stamp: "Formatted at 2026-09-28 13:08:21 on dist-test-slave-p1lc"
server_key: "c335e589df4e74deecc838aad563a434"
server_key_iv: "756d0dad81f72f11aeeb7fa8ea447ecc"
server_key_version: "encryptionkey@0"
I20260928 13:08:21.749467 14614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:08:21.750383 14629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:21.750715 14614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:21.750849 14614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "c81d9bc77ee54b4c8d04746984c5ed6e"
format_stamp: "Formatted at 2026-09-28 13:08:21 on dist-test-slave-p1lc"
server_key: "c335e589df4e74deecc838aad563a434"
server_key_iv: "756d0dad81f72f11aeeb7fa8ea447ecc"
server_key_version: "encryptionkey@0"
I20260928 13:08:21.750963 14614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:21.779666 14614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:21.782228 14614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:21.782407 14614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:21.786738 14614 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:43479
I20260928 13:08:21.786783 14681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:43479 every 8 connection(s)
I20260928 13:08:21.787196 14614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:21.788311 14682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:21.790943 14682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e: Bootstrap starting.
I20260928 13:08:21.791584 14682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:21.791877 14682 log.cc:824] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:21.792672 14682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e: No bootstrap required, opened a new log
I20260928 13:08:21.794426 14682 raft_consensus.cc:348] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } }
I20260928 13:08:21.794587 14682 raft_consensus.cc:374] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:21.794611 14682 raft_consensus.cc:729] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c81d9bc77ee54b4c8d04746984c5ed6e, State: Initialized, Role: FOLLOWER
I20260928 13:08:21.794775 14682 consensus_queue.cc:260] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } }
I20260928 13:08:21.794857 14682 raft_consensus.cc:388] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:21.794881 14682 raft_consensus.cc:482] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:21.794911 14682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:21.795897 14682 raft_consensus.cc:504] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } }
I20260928 13:08:21.795995 14682 leader_election.cc:302] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c81d9bc77ee54b4c8d04746984c5ed6e; no voters:
I20260928 13:08:21.796185 14682 leader_election.cc:288] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:21.796237 14685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:21.796439 14685 raft_consensus.cc:686] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [term 1 LEADER]: Becoming Leader. State: Replica: c81d9bc77ee54b4c8d04746984c5ed6e, State: Running, Role: LEADER
I20260928 13:08:21.796499 14682 sys_catalog.cc:564] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:21.796556 14685 consensus_queue.cc:237] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } }
I20260928 13:08:21.796949 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 14614
I20260928 13:08:21.796953 14687 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c81d9bc77ee54b4c8d04746984c5ed6e. Latest consensus state: current_term: 1 leader_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } }
I20260928 13:08:21.797039 14687 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:21.797058 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:21.797250 14692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:21.797305 8546 external_mini_cluster.cc:1428] Setting key e91fcfa3f5645ef4c6e21280ff498e1e
I20260928 13:08:21.797865 14692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:21.798377 14686 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } }
I20260928 13:08:21.798480 14686 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:21.799664 14692 catalog_manager.cc:1347] Generated new cluster ID: f1087f1fd83d400ab3e8de7dba38b4c0
I20260928 13:08:21.799713 14692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:08:21 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
I20260928 13:08:21.805127 14692 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:21.805727 14692 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:21.810201 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:21.797998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.8.88.190:43479)
0928 13:08:21.798178 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:21.798188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:21.798657 (+ 469us) server_negotiation.cc:407] Connection header received
0928 13:08:21.798783 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:21.798790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:21.798849 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:21.798979 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:21.800203 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:21.801290 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:21.802052 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:21.802508 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:21.802636 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:21.805318 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:21.805323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:21.805324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:21.805337 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:21.808336 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:21.808782 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:21.808786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:21.808786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:21.808944 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:21.809173 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:21.809174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:21.809174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:21.809499 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:21.809668 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:21.809753 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 13:08:21.809943 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260928 13:08:21.810439 14692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e: Generated new TSK 0
I20260928 13:08:21.810691 14692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab.
Entry for principal oryx/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab.
I20260928 13:08:21.844861 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:43479
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab
--principal=oryx/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:21.933817 14708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:21.934077 14708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:21.934105 14708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:21.935442 14708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:21.935499 14708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:21.935528 14708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:08:21.937204 14708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab
--principal=oryx/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:43479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:21.937492 14708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:21.937865 14708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:21.944340 14714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:21.944326 14716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:21.944326 14713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:21.945041 14708 server_base.cc:1034] running on GCE node
I20260928 13:08:21.945309 14708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:21.945636 14708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:21.946812 14708 hybrid_clock.cc:648] HybridClock initialized: now 1790600901946792 us; error 36 us; skew 500 ppm
Sep 28 13:08:21 dist-test-slave-p1lc krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600901, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:21.949838 14708 init.cc:377] Logged in from keytab as oryx/127.8.88.129@KRBTEST.COM (short username oryx)
I20260928 13:08:21.950446 14708 webserver.cc:460] Webserver started at http://127.8.88.129:44747/ using document root <none> and password file <none>
I20260928 13:08:21.950724 14708 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:21.950781 14708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:21.950908 14708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:21.952337 14708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "0ed8b333e4314457a4364a01642d9da0"
format_stamp: "Formatted at 2026-09-28 13:08:21 on dist-test-slave-p1lc"
server_key: "80c716c09b98636cd8f3cd7a229a2b9d"
server_key_iv: "3ef69b4774454ba5b2772fd88e49ec7e"
server_key_version: "encryptionkey@0"
I20260928 13:08:21.952759 14708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "0ed8b333e4314457a4364a01642d9da0"
format_stamp: "Formatted at 2026-09-28 13:08:21 on dist-test-slave-p1lc"
server_key: "80c716c09b98636cd8f3cd7a229a2b9d"
server_key_iv: "3ef69b4774454ba5b2772fd88e49ec7e"
server_key_version: "encryptionkey@0"
I20260928 13:08:21.954577 14708 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 13:08:21.955550 14723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:21.955802 14708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:21.955919 14708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "0ed8b333e4314457a4364a01642d9da0"
format_stamp: "Formatted at 2026-09-28 13:08:21 on dist-test-slave-p1lc"
server_key: "80c716c09b98636cd8f3cd7a229a2b9d"
server_key_iv: "3ef69b4774454ba5b2772fd88e49ec7e"
server_key_version: "encryptionkey@0"
I20260928 13:08:21.956002 14708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:21.972621 14708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:21.975108 14708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:21.975242 14708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:21.975503 14708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:21.975832 14708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:21.975867 14708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:21.975890 14708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:21.975903 14708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:21.982110 14708 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:44799
I20260928 13:08:21.982178 14836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:44799 every 8 connection(s)
I20260928 13:08:21.982538 14708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
Sep 28 13:08:21 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600901, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
I20260928 13:08:21.991050 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 14708
I20260928 13:08:21.991150 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:08:21.991350 8546 external_mini_cluster.cc:1428] Setting key aaed3ceab1b24946f2d9e75008b001b7
I20260928 13:08:21.993126 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:21.983501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:41731 (local address 127.8.88.190:43479)
0928 13:08:21.983578 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:21.983581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:21.983742 (+ 161us) server_negotiation.cc:407] Connection header received
0928 13:08:21.984332 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:21.984333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:21.984350 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:21.984528 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:21.985600 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:21.986137 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:21.987181 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:21.987415 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:21.987491 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:21.989860 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:21.989862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:21.989862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:21.989870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:21.991280 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:21.991850 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:21.991851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:21.991852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:21.991916 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:21.992230 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:21.992232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:21.992232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:21.992398 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:21.992483 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:21.992811 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 13:08:21.992952 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 13:08:21.993223 14839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:21.983345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:41731)
0928 13:08:21.983606 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:21.983659 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:21.984220 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:21.984610 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:21.984619 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:21.985272 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:21.985480 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:21.985485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:21.986234 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:21.986235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:21.987084 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:21.987086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:21.987512 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:21.987513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:21.987640 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:21.988179 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:21.988188 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:21.989749 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:21.991379 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:21.991388 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:21.991390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:21.991774 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:21.991994 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:21.991995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:21.991995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:21.992152 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:21.992499 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:21.992506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:21.992656 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 13:08:21.992764 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 13:08:21.993010 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":111,"threads_started":1}
I20260928 13:08:21.993793 14837 heartbeater.cc:347] Connected to a master server at 127.8.88.190:43479
I20260928 13:08:21.993916 14837 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:21.994125 14837 heartbeater.cc:511] Master 127.8.88.190:43479 requested a full tablet report, sending...
I20260928 13:08:21.994786 14646 ts_manager.cc:194] Registered new tserver with Master: 0ed8b333e4314457a4364a01642d9da0 (127.8.88.129:44799)
I20260928 13:08:21.995476 14646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:41731
WARNING: no policy specified for oryx/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab.
Entry for principal oryx/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab.
I20260928 13:08:22.020376 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:43479
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab
--principal=oryx/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:22.105840 14842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:22.106096 14842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:22.106140 14842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:22.107406 14842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:22.107482 14842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:22.107529 14842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:08:22.109211 14842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab
--principal=oryx/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:43479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:22.109498 14842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:22.109767 14842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:22.115425 14848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:22.115419 14850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:22.115446 14842 server_base.cc:1034] running on GCE node
W20260928 13:08:22.115427 14847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:22.116020 14842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:22.116248 14842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:22.117413 14842 hybrid_clock.cc:648] HybridClock initialized: now 1790600902117401 us; error 34 us; skew 500 ppm
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600902, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:22.120198 14842 init.cc:377] Logged in from keytab as oryx/127.8.88.130@KRBTEST.COM (short username oryx)
I20260928 13:08:22.120734 14842 webserver.cc:460] Webserver started at http://127.8.88.130:37163/ using document root <none> and password file <none>
I20260928 13:08:22.121013 14842 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:22.121096 14842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:22.121258 14842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:22.122375 14842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "8756a6ddde784ad28cbdc9d1fb4183de"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "7977e7dae3ab3b541cefd31d7c20f31f"
server_key_iv: "91c7b4ea61a6c7bb0f2735f624c9edf7"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.122763 14842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "8756a6ddde784ad28cbdc9d1fb4183de"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "7977e7dae3ab3b541cefd31d7c20f31f"
server_key_iv: "91c7b4ea61a6c7bb0f2735f624c9edf7"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.124171 14842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:08:22.125124 14857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:22.125327 14842 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:22.125444 14842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "8756a6ddde784ad28cbdc9d1fb4183de"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "7977e7dae3ab3b541cefd31d7c20f31f"
server_key_iv: "91c7b4ea61a6c7bb0f2735f624c9edf7"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.125521 14842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:22.145238 14842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:22.147843 14842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:22.148041 14842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:22.148299 14842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:22.148643 14842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:22.148696 14842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:22.148746 14842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:22.148777 14842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:22.155349 14842 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:41485
I20260928 13:08:22.155507 14970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:41485 every 8 connection(s)
I20260928 13:08:22.155829 14842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600902, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
I20260928 13:08:22.165771 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 14842
I20260928 13:08:22.165905 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:08:22.166184 8546 external_mini_cluster.cc:1428] Setting key 535dcdf0c981117e36c5f937560ad935
I20260928 13:08:22.166622 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.156787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:59279 (local address 127.8.88.190:43479)
0928 13:08:22.156841 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.156849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.157026 (+ 177us) server_negotiation.cc:407] Connection header received
0928 13:08:22.157711 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.157712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.157725 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.157798 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.158879 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.159409 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.160727 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.161022 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.161089 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.163526 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.163527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.163528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.163535 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.165078 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.165517 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.165518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.165518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.165599 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.165856 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.165857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.165857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.166029 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.166116 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.166443 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.166526 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:08:22.166920 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.156635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:59279)
0928 13:08:22.156894 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.156943 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.157453 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.157908 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.157912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.158546 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.158763 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.158769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.159528 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.159529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.160622 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.160625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.161127 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.161128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.161241 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.161875 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.161884 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.163334 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.165151 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.165157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.165159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.165384 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.165670 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.165671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.165671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.165777 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.166115 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.166128 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.166261 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.166412 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.166667 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
I20260928 13:08:22.167368 14971 heartbeater.cc:347] Connected to a master server at 127.8.88.190:43479
I20260928 13:08:22.167459 14971 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:22.167665 14971 heartbeater.cc:511] Master 127.8.88.190:43479 requested a full tablet report, sending...
I20260928 13:08:22.168079 14646 ts_manager.cc:194] Registered new tserver with Master: 8756a6ddde784ad28cbdc9d1fb4183de (127.8.88.130:41485)
I20260928 13:08:22.168483 14646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:59279
WARNING: no policy specified for oryx/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab.
Entry for principal oryx/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab.
I20260928 13:08:22.193418 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:43479
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab
--principal=oryx/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:22.278910 14976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:22.279148 14976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:22.279175 14976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:22.280390 14976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:22.280435 14976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:22.280463 14976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:08:22.281934 14976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab
--principal=oryx/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:43479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:22.282141 14976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:22.282328 14976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:22.287727 14982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:22.287707 14984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:22.287707 14981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:22.288455 14976 server_base.cc:1034] running on GCE node
I20260928 13:08:22.288707 14976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:22.289017 14976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:22.290196 14976 hybrid_clock.cc:648] HybridClock initialized: now 1790600902290198 us; error 46 us; skew 500 ppm
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600902, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:22.293118 14976 init.cc:377] Logged in from keytab as oryx/127.8.88.131@KRBTEST.COM (short username oryx)
I20260928 13:08:22.293637 14976 webserver.cc:460] Webserver started at http://127.8.88.131:43263/ using document root <none> and password file <none>
I20260928 13:08:22.293875 14976 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:22.293942 14976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:22.294097 14976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:22.295176 14976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "503b2cb44b85450e92f6ca9d3b086f43"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "cf8b9c8ca02bb8a4dea6cef685b0fcdf"
server_key_iv: "ad26532a5c72e3087e333ec2d67c891c"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.295552 14976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "503b2cb44b85450e92f6ca9d3b086f43"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "cf8b9c8ca02bb8a4dea6cef685b0fcdf"
server_key_iv: "ad26532a5c72e3087e333ec2d67c891c"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.297063 14976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:08:22.297958 14991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:22.298314 14976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:22.298414 14976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "503b2cb44b85450e92f6ca9d3b086f43"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "cf8b9c8ca02bb8a4dea6cef685b0fcdf"
server_key_iv: "ad26532a5c72e3087e333ec2d67c891c"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.298498 14976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:22.312578 14976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:22.314778 14976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:22.314942 14976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:22.315191 14976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:22.315512 14976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:22.315564 14976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:22.315605 14976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:22.315637 14976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:22.322167 14976 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:45765
I20260928 13:08:22.322252 15104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:45765 every 8 connection(s)
I20260928 13:08:22.322592 14976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:08:22.328830 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 14976
I20260928 13:08:22.328959 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600902, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
I20260928 13:08:22.329181 8546 external_mini_cluster.cc:1428] Setting key e5a1b6a68a01928ef48ce4dcaf9ad6f5
I20260928 13:08:22.332828 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.323330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:53457 (local address 127.8.88.190:43479)
0928 13:08:22.323400 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.323401 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.323610 (+ 209us) server_negotiation.cc:407] Connection header received
0928 13:08:22.324152 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.324153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.324167 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.324224 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.325277 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.325789 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.326818 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.327036 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.327122 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.329814 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.329816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.329817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.329826 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.331374 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.331796 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.331797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.331797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.331858 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.332193 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.332194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.332194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.332323 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.332388 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.332689 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.332763 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 13:08:22.333096 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.323321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:53457)
0928 13:08:22.323495 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.323532 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.324046 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.324308 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.324312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.324964 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.325174 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.325178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.325883 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.325884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.326717 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.326719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.327131 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.327132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.327260 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.327777 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.327786 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.329735 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.331450 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.331456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.331458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.331693 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.331954 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.331955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.331955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.332103 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.332407 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.332413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.332557 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.332657 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.332795 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260928 13:08:22.333667 15105 heartbeater.cc:347] Connected to a master server at 127.8.88.190:43479
I20260928 13:08:22.333778 15105 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:22.333985 15105 heartbeater.cc:511] Master 127.8.88.190:43479 requested a full tablet report, sending...
I20260928 13:08:22.334389 14646 ts_manager.cc:194] Registered new tserver with Master: 503b2cb44b85450e92f6ca9d3b086f43 (127.8.88.131:45765)
I20260928 13:08:22.334846 14646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:53457
I20260928 13:08:22.341245 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.88.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.88.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.88.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.189.keytab.
Entry for principal oryx/127.8.88.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.189.keytab.
I20260928 13:08:22.369050 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.189:37155
--webserver_interface=127.8.88.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.189.keytab
--principal=oryx/127.8.88.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:43479,127.8.88.189:37155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:22.454790 15110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:22.455005 15110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:22.455029 15110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:22.456173 15110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:22.456214 15110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:22.456228 15110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:22.456240 15110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:22.456252 15110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:22.457988 15110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:43479,127.8.88.189:37155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.189.keytab
--principal=oryx/127.8.88.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.189:37155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.88.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
W20260928 13:08:22.458101 15110 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.88.190:43479,127.8.88.189:37155'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 13:08:22.458223 15110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:22.458405 15110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:22.463547 15118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:22.463583 15115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:22.463510 15116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:22.463752 15110 server_base.cc:1034] running on GCE node
I20260928 13:08:22.464056 15110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:22.464371 15110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:22.465525 15110 hybrid_clock.cc:648] HybridClock initialized: now 1790600902465538 us; error 67 us; skew 500 ppm
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600902, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:22.468124 15110 init.cc:377] Logged in from keytab as oryx/127.8.88.189@KRBTEST.COM (short username oryx)
I20260928 13:08:22.468609 15110 webserver.cc:460] Webserver started at http://127.8.88.189:38087/ using document root <none> and password file <none>
I20260928 13:08:22.468822 15110 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:22.468911 15110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:22.469029 15110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:22.470068 15110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/data/instance:
uuid: "81409d6217b64c2794dca75da4b7e534"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "25538f61a479dcf2b7b7d86c8ec268bb"
server_key_iv: "098d1587bed6d7675bc25cf04c699c9c"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.470389 15110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/wal/instance:
uuid: "81409d6217b64c2794dca75da4b7e534"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "25538f61a479dcf2b7b7d86c8ec268bb"
server_key_iv: "098d1587bed6d7675bc25cf04c699c9c"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.471773 15110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:08:22.472512 15125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:22.472702 15110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:22.472792 15110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/wal
uuid: "81409d6217b64c2794dca75da4b7e534"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "25538f61a479dcf2b7b7d86c8ec268bb"
server_key_iv: "098d1587bed6d7675bc25cf04c699c9c"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.472859 15110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:22.484352 15110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:22.486548 15110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:22.486666 15110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:22.490612 15110 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.189:37155
I20260928 13:08:22.490685 15177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.189:37155 every 8 connection(s)
I20260928 13:08:22.491061 15110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/data/info.pb
I20260928 13:08:22.492136 15178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:22.494035 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 15110
I20260928 13:08:22.493990 15178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600902, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.189@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.88.189@KRBTEST.COM
I20260928 13:08:22.505676 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.494796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.8.88.190:43479)
0928 13:08:22.494852 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.494855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.495112 (+ 257us) server_negotiation.cc:407] Connection header received
0928 13:08:22.495620 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.495622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.495635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.495691 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.496541 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.497449 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.498866 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.499189 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.499276 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.502449 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.502450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.502451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.502459 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.504094 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.504489 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.504490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.504490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.504712 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.504994 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.504995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.504995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.505155 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.505257 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.505458 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.505553 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 13:08:22.505796 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.494806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.8.88.189:37155)
0928 13:08:22.495025 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.495047 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.495080 (+ 33us) server_negotiation.cc:407] Connection header received
0928 13:08:22.495515 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.495527 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.495557 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.495679 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.496538 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.497839 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.498938 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.499456 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.499508 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.502387 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.502394 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.502394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.502401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.503988 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.504363 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.504366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.504367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.504465 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.504776 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.504776 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.504777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.505147 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.505297 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.505381 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.505558 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":179,"thread_start_us":117,"threads_started":1}
I20260928 13:08:22.505796 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.494806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:52162)
0928 13:08:22.494998 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.495038 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.495497 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.495737 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.495740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.496174 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.496447 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.496454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.497550 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.497551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.498765 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.498768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.499292 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.499294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.499415 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.500095 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.500107 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.502334 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.504176 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.504181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.504183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.504410 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.504779 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.504780 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.504780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.504924 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.505257 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.505262 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.505324 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.505417 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.505608 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":92,"thread_start_us":108,"threads_started":1}
I20260928 13:08:22.506562 15178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600902, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.189@KRBTEST.COM for oryx/127.8.88.189@KRBTEST.COM
I20260928 13:08:22.512638 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.506922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.8.88.189:37155)
0928 13:08:22.506995 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.506997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.507003 (+ 6us) server_negotiation.cc:407] Connection header received
0928 13:08:22.507071 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.507072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.507079 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.507131 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.507349 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.507811 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.508477 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.508689 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.508749 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.510585 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.510586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.510587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.510593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.511749 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.512065 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.512065 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.512066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.512113 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.512273 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.512274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.512274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.512365 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.512412 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.512513 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.512575 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 13:08:22.512764 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.506834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37155 (local address 127.0.0.1:36210)
0928 13:08:22.506953 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.506966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.507011 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.507133 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.507133 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.507250 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.507299 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.507300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.507893 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.507894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.508388 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.508390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.508756 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.508757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.508812 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.509271 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.509275 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.510467 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.511817 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.511818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.511818 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.511979 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.512170 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.512170 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.512170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.512222 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.512413 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.512414 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.512449 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.512505 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.512630 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":42}
I20260928 13:08:22.514204 15178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: Bootstrap starting.
I20260928 13:08:22.515048 15178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:22.515321 15178 log.cc:824] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:22.516132 15178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: No bootstrap required, opened a new log
I20260928 13:08:22.517853 15178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } }
I20260928 13:08:22.518013 15178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:22.518069 15178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81409d6217b64c2794dca75da4b7e534, State: Initialized, Role: FOLLOWER
I20260928 13:08:22.518219 15178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } }
I20260928 13:08:22.518573 15178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:22.518558 15184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } } }
I20260928 13:08:22.518682 15184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 13:08:22.520277 15195 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:08:22.520377 15195 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:08:22.522006 15110 master_runner.cc:428] Detected that this master 81409d6217b64c2794dca75da4b7e534 is joining an existing cluster
I20260928 13:08:22.522074 15110 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.88.189:37155
I20260928 13:08:22.530634 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.523318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.8.88.190:43479)
0928 13:08:22.523367 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.523369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.523485 (+ 116us) server_negotiation.cc:407] Connection header received
0928 13:08:22.523591 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.523592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.523604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.523634 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.523997 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.524484 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.525709 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.525920 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.526008 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.527139 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.527141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.527141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.527148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.529199 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.529799 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.529800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.529801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.529857 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.530076 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.530078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.530078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.530200 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.530245 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.530445 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.530529 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 13:08:22.530750 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.523090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36222 (local address 127.8.88.189:37155)
0928 13:08:22.523131 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.523133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.523247 (+ 114us) server_negotiation.cc:407] Connection header received
0928 13:08:22.523333 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.523334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.523343 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.523373 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.523870 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.524644 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.525692 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.525995 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.526066 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.527276 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.527277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.527277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.527283 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.529378 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.529766 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.529767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.529767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.529825 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.530059 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.530060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.530060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.530197 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.530266 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.530460 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.530610 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":19}
I20260928 13:08:22.530685 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.523317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:52170)
0928 13:08:22.523429 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.523439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.523508 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.523699 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.523700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.523863 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.523916 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.523917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.524570 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.524572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.525596 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.525598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.526020 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.526021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.526081 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.526590 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.526595 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.527024 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.529286 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.529287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.529288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.529699 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.529931 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.529931 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.529931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.529982 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.530294 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.530294 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.530339 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.530416 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.530545 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260928 13:08:22.531371 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.523048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37155 (local address 127.0.0.1:36222)
0928 13:08:22.523175 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.523185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.523263 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.523433 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.523434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.523678 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.523783 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.523785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.524728 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.524729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.525596 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.525598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.526116 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.526117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.526178 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.526742 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.526747 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.527193 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.529454 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.529455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.529455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.529675 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.529890 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.529890 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.529890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.529957 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.530273 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.530274 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.530341 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.530488 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.531262 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":7,"thread_start_us":79,"threads_started":1}
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600901, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.190@KRBTEST.COM for oryx/127.8.88.189@KRBTEST.COM
I20260928 13:08:22.541680 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.533649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37155 (local address 127.0.0.1:36234)
0928 13:08:22.533793 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.533837 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.533910 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.534052 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.534054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.534323 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.534375 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.534377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.535292 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.535293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.535917 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.535920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.536379 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.536380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.536442 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.537002 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.537007 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.538614 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.540459 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.540466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.540468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.540690 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.541002 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.541004 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.541004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.541068 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.541350 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.541356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.541410 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.541480 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.541587 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":94,"threads_started":1}
I20260928 13:08:22.541706 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.533675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.8.88.189:37155)
0928 13:08:22.533715 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.533718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.533900 (+ 182us) server_negotiation.cc:407] Connection header received
0928 13:08:22.533972 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.533972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.533982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.534049 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.534444 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.535194 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.535999 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.536305 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.536376 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.538707 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.538709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.538709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.538718 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.540368 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.540765 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.540766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.540766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.540832 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.541138 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.541139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.541140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.541283 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.541354 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.541520 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.541610 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:08:22.542335 14646 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.88.189:37155: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "81409d6217b64c2794dca75da4b7e534"
member_type: NON_VOTER
last_known_addr {
host: "127.8.88.189"
port: 37155
}
attrs {
promote: true
}
}
dest_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e"
cas_config_opid_index: -1
I20260928 13:08:22.542697 14646 consensus_queue.cc:237] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true } }
I20260928 13:08:22.543208 15205 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 81409d6217b64c2794dca75da4b7e534 (127.8.88.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true } } }
I20260928 13:08:22.543573 15152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:22.544286 15205 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.88.189:37155
I20260928 13:08:22.544390 15152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Refusing update from remote peer c81d9bc77ee54b4c8d04746984c5ed6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 13:08:22.544516 15110 master.cc:537] Master@127.8.88.189:37155 shutting down...
I20260928 13:08:22.544790 15204 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true } } }
I20260928 13:08:22.544881 15204 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:22.544823 15205 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 13:08:22.545642 14631 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 13:08:22.546094 14631 consensus_peers.cc:487] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e -> Peer 81409d6217b64c2794dca75da4b7e534 (127.8.88.189:37155): Couldn't send request to peer 81409d6217b64c2794dca75da4b7e534. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 13:08:22.547206 15110 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:22.547297 15110 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:22.547345 15110 tablet_replica.cc:330] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: stopping tablet replica
I20260928 13:08:22.549752 15110 master.cc:552] Master@127.8.88.189:37155 shutdown complete.
I20260928 13:08:22.549818 15110 master_runner.cc:305] Clearing existing system tablet
I20260928 13:08:22.549954 15110 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:22.550014 15110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:22.550663 15110 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:22.551129 15209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:22.551292 15110 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:22.551352 15110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:22.551376 15110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/wal
uuid: "81409d6217b64c2794dca75da4b7e534"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "25538f61a479dcf2b7b7d86c8ec268bb"
server_key_iv: "098d1587bed6d7675bc25cf04c699c9c"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.552069 15110 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:22.553544 15110 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 13:08:22.553608 15110 log.cc:1196] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 13:08:22.553864 15110 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: Deleting consensus metadata
I20260928 13:08:22.554039 15110 master_runner.cc:315] Copying system tablet from 127.8.88.190:43479
I20260928 13:08:22.554734 15110 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: tablet copy: Beginning tablet copy session from remote peer at address 127.8.88.190:43479
I20260928 13:08:22.562837 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.555179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.8.88.190:43479)
0928 13:08:22.555246 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.555248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.555315 (+ 67us) server_negotiation.cc:407] Connection header received
0928 13:08:22.555396 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.555397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.555410 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.555474 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.555778 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.556612 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.557878 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.558195 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.558279 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.559537 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.559539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.559540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.559548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.561390 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.561836 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.561837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.561838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.561904 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.562237 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.562238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.562239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.562381 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.562458 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.562691 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.562769 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 13:08:22.562872 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.555093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:52172)
0928 13:08:22.555248 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.555259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.555324 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.555478 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.555478 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.555643 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.555701 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.555702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.556981 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.556983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.557780 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.557783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.558303 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.558304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.558369 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.558928 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.558934 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.559426 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.561525 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.561527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.561527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.561753 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.562016 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.562018 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.562018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.562089 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.562463 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.562464 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.562536 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.562612 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.562743 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260928 13:08:22.563226 14666 tablet_copy_service.cc:140] P c81d9bc77ee54b4c8d04746984c5ed6e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 81409d6217b64c2794dca75da4b7e534 ({username='oryx', principal='oryx/127.8.88.189@KRBTEST.COM'} at 127.0.0.1:52172)
I20260928 13:08:22.563359 14666 tablet_copy_service.cc:161] P c81d9bc77ee54b4c8d04746984c5ed6e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 81409d6217b64c2794dca75da4b7e534 at {username='oryx', principal='oryx/127.8.88.189@KRBTEST.COM'} at 127.0.0.1:52172: session id = 81409d6217b64c2794dca75da4b7e534-00000000000000000000000000000000
I20260928 13:08:22.564250 14666 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 13:08:22.565095 15110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 13:08:22.566152 15110 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: tablet copy: Starting download of 0 data blocks...
I20260928 13:08:22.566320 15110 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: tablet copy: Starting download of 1 WAL segments...
I20260928 13:08:22.567328 15110 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 13:08:22.568300 14666 tablet_copy_service.cc:342] P c81d9bc77ee54b4c8d04746984c5ed6e: Request end of tablet copy session 81409d6217b64c2794dca75da4b7e534-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.88.189@KRBTEST.COM'} at 127.0.0.1:52172
I20260928 13:08:22.568367 14666 tablet_copy_service.cc:433] P c81d9bc77ee54b4c8d04746984c5ed6e: ending tablet copy session 81409d6217b64c2794dca75da4b7e534-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 81409d6217b64c2794dca75da4b7e534
I20260928 13:08:22.571007 15110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:22.572448 15216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:22.572731 15217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:22.572897 15219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:22.573024 15110 server_base.cc:1034] running on GCE node
I20260928 13:08:22.573150 15110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:22.573305 15110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:22.574488 15110 hybrid_clock.cc:648] HybridClock initialized: now 1790600902574478 us; error 31 us; skew 500 ppm
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600902, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:22.576148 15110 init.cc:377] Logged in from keytab as oryx/127.8.88.189@KRBTEST.COM (short username oryx)
I20260928 13:08:22.576453 15110 webserver.cc:460] Webserver started at http://127.8.88.189:38393/ using document root <none> and password file <none>
I20260928 13:08:22.576586 15110 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:22.576627 15110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:22.577345 15110 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:22.577911 15226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:22.578097 15110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 13:08:22.578168 15110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/wal
uuid: "81409d6217b64c2794dca75da4b7e534"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "25538f61a479dcf2b7b7d86c8ec268bb"
server_key_iv: "098d1587bed6d7675bc25cf04c699c9c"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.578234 15110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:22.603416 15110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:22.605139 15110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:22.608168 15110 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.189:37155
I20260928 13:08:22.608253 15278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.189:37155 every 8 connection(s)
I20260928 13:08:22.608527 15110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-1/data/info.pb
I20260928 13:08:22.609515 15279 sys_catalog.cc:263] Verifying existing consensus state
I20260928 13:08:22.610056 15279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: Bootstrap starting.
I20260928 13:08:22.613164 15279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:08:22.613423 15279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: Bootstrap complete.
I20260928 13:08:22.613675 15279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true } }
I20260928 13:08:22.613771 15279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 81409d6217b64c2794dca75da4b7e534, State: Initialized, Role: LEARNER
I20260928 13:08:22.613813 15279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true } }
I20260928 13:08:22.613989 15279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:22.613994 15282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true } } }
I20260928 13:08:22.614178 15282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [sys.catalog]: This master's current role is: LEARNER
I20260928 13:08:22.615496 15293 catalog_manager.cc:1259] Loaded cluster ID: f1087f1fd83d400ab3e8de7dba38b4c0
I20260928 13:08:22.615653 15293 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: loading cluster ID for follower catalog manager: success
I20260928 13:08:22.616585 15293 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: acquiring CA information for follower catalog manager: success
I20260928 13:08:22.616918 15293 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:08:22.664958 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.658174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36262 (local address 127.8.88.189:37155)
0928 13:08:22.658320 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.658321 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.658328 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:08:22.658351 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.658352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.658361 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.658433 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.659005 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.659868 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.660672 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.660930 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.661007 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.662337 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.662339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.662339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.662348 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.663788 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.664148 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.664149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.664150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.664208 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.664400 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.664400 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.664401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.664518 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.664644 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.664734 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.664818 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for oryx/127.8.88.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.88.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.88.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.188.keytab.
Entry for principal oryx/127.8.88.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.188.keytab.
I20260928 13:08:22.693657 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.188:45571
--webserver_interface=127.8.88.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.188.keytab
--principal=oryx/127.8.88.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:43479,127.8.88.189:37155,127.8.88.188:45571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:22.780715 15297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:22.780992 15297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:22.781038 15297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:22.782382 15297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:22.782450 15297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:22.782474 15297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:22.782505 15297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:22.782536 15297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:22.784157 15297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:43479,127.8.88.189:37155,127.8.88.188:45571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.188.keytab
--principal=oryx/127.8.88.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.188:45571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.88.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:22.784453 15297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:22.784698 15297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:22.790859 15302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:22.790858 15305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:22.790865 15303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:22.791229 15297 server_base.cc:1034] running on GCE node
I20260928 13:08:22.791458 15297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:22.791819 15297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:22.792943 15297 hybrid_clock.cc:648] HybridClock initialized: now 1790600902792923 us; error 38 us; skew 500 ppm
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600902, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:22.795650 15297 init.cc:377] Logged in from keytab as oryx/127.8.88.188@KRBTEST.COM (short username oryx)
I20260928 13:08:22.796265 15297 webserver.cc:460] Webserver started at http://127.8.88.188:45935/ using document root <none> and password file <none>
I20260928 13:08:22.796506 15297 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:22.796559 15297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:22.796663 15297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:22.798028 15297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/data/instance:
uuid: "5626e470e9c945789fb7f753593335a0"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "6c1d0f5098cac3cc306ed2529cce1865"
server_key_iv: "bf72d413f0cbf0cb3720cd74dead468a"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.798447 15297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/wal/instance:
uuid: "5626e470e9c945789fb7f753593335a0"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "6c1d0f5098cac3cc306ed2529cce1865"
server_key_iv: "bf72d413f0cbf0cb3720cd74dead468a"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.799939 15297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:08:22.800740 15312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:22.801087 15297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:22.801203 15297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/wal
uuid: "5626e470e9c945789fb7f753593335a0"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "6c1d0f5098cac3cc306ed2529cce1865"
server_key_iv: "bf72d413f0cbf0cb3720cd74dead468a"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.801285 15297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:22.817932 15297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:22.820362 15297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:22.820518 15297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:22.825692 15297 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.188:45571
I20260928 13:08:22.825728 15364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.188:45571 every 8 connection(s)
I20260928 13:08:22.826102 15297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/data/info.pb
I20260928 13:08:22.827123 15365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:22.829191 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 15297
I20260928 13:08:22.829175 15365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600902, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.188@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.88.188@KRBTEST.COM
I20260928 13:08:22.839010 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.830098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52176 (local address 127.8.88.190:43479)
0928 13:08:22.830158 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.830160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.830280 (+ 120us) server_negotiation.cc:407] Connection header received
0928 13:08:22.830678 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.830680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.830693 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.830755 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.831377 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.831921 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.832866 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.833107 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.833191 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.835625 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.835627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.835628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.835634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.837384 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.837886 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.837888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.837888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.837955 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.838246 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.838248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.838248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.838391 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.838473 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.838823 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.838912 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 13:08:22.839293 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.829942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:52176)
0928 13:08:22.830154 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.830198 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.830543 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.830778 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.830781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.831105 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.831278 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.831283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.832043 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.832044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.832778 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.832780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.833200 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.833200 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.833337 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.834024 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.834039 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.835528 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.837488 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.837495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.837497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.837807 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.838032 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.838033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.838034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.838146 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.838471 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.838477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.838680 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.838799 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.838967 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260928 13:08:22.839432 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.829831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55002 (local address 127.8.88.188:45571)
0928 13:08:22.830033 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.830043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.830075 (+ 32us) server_negotiation.cc:407] Connection header received
0928 13:08:22.830519 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.830526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.830547 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.830675 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.831290 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.832118 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.832853 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.833208 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.833338 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.836153 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.836160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.836160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.836168 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.838022 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.838377 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.838381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.838381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.838441 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.838632 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.838633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.838633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.838806 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.838991 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.839045 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.839320 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260928 13:08:22.840142 15365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600902, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.188@KRBTEST.COM for oryx/127.8.88.189@KRBTEST.COM
I20260928 13:08:22.847414 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.840535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.8.88.189:37155)
0928 13:08:22.840586 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.840588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.840594 (+ 6us) server_negotiation.cc:407] Connection header received
0928 13:08:22.840616 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.840617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.840627 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.840676 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.840981 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.841509 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.842372 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.842600 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.842668 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.844475 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.844476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.844477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.844484 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.846418 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.846784 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.846785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.846785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.846836 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.846980 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.846980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.846980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.847104 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.847150 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.847276 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.847345 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 13:08:22.847415 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.840414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37155 (local address 127.0.0.1:36272)
0928 13:08:22.840460 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.840471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.840513 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.840691 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.840692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.840809 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.840848 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.840849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.841596 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.841597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.842284 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.842287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.842676 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.842677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.842736 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.843163 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.843167 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.844394 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.846514 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.846516 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.846516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.846717 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.846889 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.846890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.846890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.846942 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.847152 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.847153 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.847193 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.847261 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.847345 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 13:08:22.847944 15365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.188" port: 45571 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600902, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.188@KRBTEST.COM for oryx/127.8.88.188@KRBTEST.COM
I20260928 13:08:22.854591 15368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.848331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:45571 (local address 127.0.0.1:55004)
0928 13:08:22.848420 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.848432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.848483 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.848602 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.848603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.848750 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.848782 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.848783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.849486 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.849487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.849966 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.849967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.850349 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.850350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.850397 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.850849 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.850853 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.852040 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.853521 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.853522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.853522 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.853733 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.853926 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.853926 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.853927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.853993 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.854243 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.854244 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.854282 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.854346 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.854466 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":4}
I20260928 13:08:22.854591 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.848368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.8.88.188:45571)
0928 13:08:22.848431 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.848434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.848469 (+ 35us) server_negotiation.cc:407] Connection header received
0928 13:08:22.848528 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.848529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.848538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.848599 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.848859 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.849393 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.850055 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.850254 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.850346 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.852125 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.852126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.852127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.852134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.853429 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.853805 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.853806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.853807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.853864 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.854058 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.854058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.854059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.854179 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.854252 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.854371 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.854512 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":40}
I20260928 13:08:22.856272 15365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: Bootstrap starting.
I20260928 13:08:22.857226 15365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:22.857466 15365 log.cc:824] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:22.858309 15365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: No bootstrap required, opened a new log
I20260928 13:08:22.860203 15365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 45571 } }
I20260928 13:08:22.860436 15365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:22.860474 15365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5626e470e9c945789fb7f753593335a0, State: Initialized, Role: FOLLOWER
I20260928 13:08:22.860615 15365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 45571 } }
I20260928 13:08:22.861027 15365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:22.861076 15370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 45571 } } }
I20260928 13:08:22.861240 15370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 13:08:22.862915 15381 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:08:22.863034 15381 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:08:22.866717 15297 master_runner.cc:428] Detected that this master 5626e470e9c945789fb7f753593335a0 is joining an existing cluster
I20260928 13:08:22.866787 15297 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.88.188:45571
I20260928 13:08:22.876027 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.868028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.8.88.189:37155)
0928 13:08:22.868075 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.868077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.868649 (+ 572us) server_negotiation.cc:407] Connection header received
0928 13:08:22.868718 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.868719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.868735 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.868797 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.869385 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.870186 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.871075 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.871291 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.871365 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.872824 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.872826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.872826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.872835 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.874660 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.875108 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.875109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.875109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.875168 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.875405 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.875405 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.875406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.875547 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.875591 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.875838 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.875923 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:08:22.876945 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.868007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37155 (local address 127.0.0.1:36274)
0928 13:08:22.868582 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.868594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.868661 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.868846 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.868847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.869148 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.869243 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.869245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.870288 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.870288 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.870983 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.870986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.871385 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.871386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.871447 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.872289 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.872299 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.872725 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.874774 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.874775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.874776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.875021 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.875252 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.875253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.875253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.875326 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.875638 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.875639 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.875683 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.876696 (+ 1013us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.876815 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":305,"threads_started":1}
I20260928 13:08:22.879617 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.869467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.8.88.188:45571)
0928 13:08:22.869563 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.869565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.873237 (+ 3672us) server_negotiation.cc:407] Connection header received
0928 13:08:22.873317 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.873318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.873327 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.873353 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.873830 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.874548 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.875651 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.875930 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.875968 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.877069 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.877070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.877070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.877077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.878485 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.878820 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.878821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.878822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.878874 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.879092 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.879093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.879093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.879205 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.879277 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.879422 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.879509 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 13:08:22.879701 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.868467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:45571 (local address 127.0.0.1:55006)
0928 13:08:22.873187 (+ 4720us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.873196 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.873256 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.873407 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.873408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.873662 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.873748 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.873750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.874637 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.874637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.875558 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.875560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.876039 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.876039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.876099 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.876526 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.876530 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.876990 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.878574 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.878574 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.878575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.878752 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.878958 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.878959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.878959 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.879034 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.879268 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.879269 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.879319 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.879400 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.879561 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4704,"mutex_wait_us":97,"thread_start_us":65,"threads_started":1}
I20260928 13:08:22.891390 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 14708
I20260928 13:08:22.897149 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.868189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.8.88.190:43479)
0928 13:08:22.868234 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.868236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.873086 (+ 4850us) server_negotiation.cc:407] Connection header received
0928 13:08:22.873152 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.873154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.873180 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.873235 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.873843 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.874581 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.875659 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.875967 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.876006 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.894053 (+ 18047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.894056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.894056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.894066 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.895633 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.896027 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.896029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.896030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.896090 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.896315 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.896316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.896317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.896672 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.896788 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.896956 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.897048 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 13:08:22.897228 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.868291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:52184)
0928 13:08:22.873013 (+ 4722us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.873024 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.873094 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.873285 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.873286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.873659 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.873745 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.873747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.874667 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.874668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.875565 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.875567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.876073 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.876074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.876127 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.893575 (+ 17448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.893581 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.893925 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.895753 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.895754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.895755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.895949 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.896164 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.896164 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.896165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.896258 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.896778 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.896779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.896833 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.896929 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.897049 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4704,"mutex_wait_us":34,"thread_start_us":67,"threads_started":1}
I20260928 13:08:22.898550 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:44799
--local_ip_for_outbound_sockets=127.8.88.129
--tserver_master_addrs=127.8.88.190:43479
--webserver_port=44747
--webserver_interface=127.8.88.129
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab
--principal=oryx/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.88.190:43479,127.8.88.189:37155
--tserver_master_addrs=127.8.88.190:43479,127.8.88.189:37155,127.8.88.188:45571 with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600901, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.190@KRBTEST.COM for oryx/127.8.88.188@KRBTEST.COM
I20260928 13:08:22.908241 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.899652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:45571 (local address 127.0.0.1:55014)
0928 13:08:22.899704 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.899714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.899749 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.899965 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.899966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.900123 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.900164 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.900165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.900797 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.900798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.901726 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.901729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.902274 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.902275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.902355 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.902934 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.902940 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.904726 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.906894 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.906896 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.906896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.907175 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.907405 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.907406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.907407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.907476 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.907807 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.907808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.907865 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.907938 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.908071 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 13:08:22.908659 15367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.899704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.8.88.188:45571)
0928 13:08:22.899743 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.899745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.899755 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:08:22.899832 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.899833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.899853 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.899904 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.900239 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.900717 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.901821 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.902152 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.902217 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.904839 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.904841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.904841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.904850 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.906778 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.907267 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.907269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.907269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.907332 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.907549 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.907550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.907550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.907707 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.907775 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.908405 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.908532 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 13:08:22.909230 14646 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.88.188:45571: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5626e470e9c945789fb7f753593335a0"
member_type: NON_VOTER
last_known_addr {
host: "127.8.88.188"
port: 45571
}
attrs {
promote: true
}
}
dest_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e"
cas_config_opid_index: 5
I20260928 13:08:22.909578 14646 consensus_queue.cc:237] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } }
I20260928 13:08:22.910202 15205 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5626e470e9c945789fb7f753593335a0 (127.8.88.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } } }
I20260928 13:08:22.911194 15205 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.88.188:45571
I20260928 13:08:22.911274 15205 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } } }
I20260928 13:08:22.911345 15205 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:22.911767 15338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:22.911971 15297 master.cc:537] Master@127.8.88.188:45571 shutting down...
I20260928 13:08:22.912863 15338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [term 1 FOLLOWER]: Refusing update from remote peer c81d9bc77ee54b4c8d04746984c5ed6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 13:08:22.913224 15204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 13:08:22.914047 14631 consensus_peers.cc:487] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e -> Peer 5626e470e9c945789fb7f753593335a0 (127.8.88.188:45571): Couldn't send request to peer 5626e470e9c945789fb7f753593335a0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 13:08:22.915324 15297 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:22.915423 15297 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:22.915467 15297 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: stopping tablet replica
I20260928 13:08:22.916481 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.911763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36276 (local address 127.8.88.189:37155)
0928 13:08:22.911805 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.911808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.912484 (+ 676us) server_negotiation.cc:407] Connection header received
0928 13:08:22.912678 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.912680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.912704 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.912746 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:22.912984 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.913721 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.915146 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.915772 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.915822 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.915895 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.916362 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.916408 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:08:22.916741 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.911663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37155 (local address 127.0.0.1:36276)
0928 13:08:22.912422 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.912435 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.912527 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.912830 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.912830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.912832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:22.912905 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.912907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.913826 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.913827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.915019 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.915022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.916174 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.916175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.916250 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.916274 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.916623 (+ 349us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.916676 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741}
I20260928 13:08:22.917065 15253 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 LEARNER]: Refusing update from remote peer c81d9bc77ee54b4c8d04746984c5ed6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 13:08:22.917268 15204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 13:08:22.917855 15253 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5626e470e9c945789fb7f753593335a0 (127.8.88.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } } }
I20260928 13:08:22.919196 15204 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e: attempting to promote NON_VOTER 81409d6217b64c2794dca75da4b7e534 to VOTER
I20260928 13:08:22.919230 15282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c81d9bc77ee54b4c8d04746984c5ed6e. Latest consensus state: current_term: 1 leader_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } } }
I20260928 13:08:22.919320 15282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [sys.catalog]: This master's current role is: LEARNER
I20260928 13:08:22.919356 15282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: NON_VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: true } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } } }
I20260928 13:08:22.919394 15282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [sys.catalog]: This master's current role is: LEARNER
I20260928 13:08:22.919415 15204 consensus_queue.cc:237] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } }
I20260928 13:08:22.920068 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.919845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:45571 (local address 127.0.0.1:55028)
0928 13:08:22.919969 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.920022 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260928 13:08:22.920076 15253 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 LEARNER]: Refusing update from remote peer c81d9bc77ee54b4c8d04746984c5ed6e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 13:08:22.920275 15205 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 13:08:22.920564 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.920468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:45571 (local address 127.0.0.1:55042)
0928 13:08:22.920515 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.920543 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:45571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 13:08:22.920712 14631 consensus_peers.cc:487] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e -> Peer 5626e470e9c945789fb7f753593335a0 (127.8.88.188:45571): Couldn't send request to peer 5626e470e9c945789fb7f753593335a0. Status: Network error: Client connection negotiation failed: client connection to 127.8.88.188:45571: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 13:08:22.921090 15204 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 81409d6217b64c2794dca75da4b7e534 (127.8.88.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } } }
I20260928 13:08:22.921244 15253 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 81409d6217b64c2794dca75da4b7e534 (127.8.88.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } } }
I20260928 13:08:22.922123 15204 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } } }
I20260928 13:08:22.922212 15204 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:22.922268 15204 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } } }
I20260928 13:08:22.922317 15204 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:22.922457 15282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } } }
I20260928 13:08:22.922540 15282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:08:22.929510 15297 master.cc:552] Master@127.8.88.188:45571 shutdown complete.
I20260928 13:08:22.929574 15297 master_runner.cc:305] Clearing existing system tablet
I20260928 13:08:22.929704 15297 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:22.929750 15297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:22.930490 15297 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:22.931046 15396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:22.931254 15297 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:22.931315 15297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:22.931339 15297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/wal
uuid: "5626e470e9c945789fb7f753593335a0"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "6c1d0f5098cac3cc306ed2529cce1865"
server_key_iv: "bf72d413f0cbf0cb3720cd74dead468a"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.932117 15297 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:22.934067 15297 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 13:08:22.934150 15297 log.cc:1196] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 13:08:22.934348 15297 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: Deleting consensus metadata
I20260928 13:08:22.934525 15297 master_runner.cc:315] Copying system tablet from 127.8.88.190:43479
I20260928 13:08:22.935166 15297 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: tablet copy: Beginning tablet copy session from remote peer at address 127.8.88.190:43479
I20260928 13:08:22.944149 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.935556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52192 (local address 127.8.88.190:43479)
0928 13:08:22.935608 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:22.935610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:22.937131 (+ 1521us) server_negotiation.cc:407] Connection header received
0928 13:08:22.937203 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:22.937204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:22.937217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:22.937291 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:22.937684 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.938446 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.939547 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.939847 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.939935 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.941190 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:22.941192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:22.941193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:22.941201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:22.942860 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.943307 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.943308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.943309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.943374 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:22.943586 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:22.943587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:22.943587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:22.943723 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:22.943788 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:22.943969 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 13:08:22.944054 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:08:22.944164 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:22.935479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:52192)
0928 13:08:22.937029 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:22.937044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:22.937135 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:22.937309 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:22.937310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:22.937532 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:22.937592 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.937594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.938568 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.938570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.939428 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:22.939431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:22.939949 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:22.939950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:22.940015 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:22.940603 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:22.940608 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:22.941096 (+ 488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:22.942969 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.942970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.942971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.943234 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.943446 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:22.943447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:22.943447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:22.943519 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:22.943795 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:22.943796 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:22.943853 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 13:08:22.943941 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 13:08:22.944058 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1520,"thread_start_us":72,"threads_started":1}
I20260928 13:08:22.944693 14666 tablet_copy_service.cc:140] P c81d9bc77ee54b4c8d04746984c5ed6e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5626e470e9c945789fb7f753593335a0 ({username='oryx', principal='oryx/127.8.88.188@KRBTEST.COM'} at 127.0.0.1:52192)
I20260928 13:08:22.944763 14666 tablet_copy_service.cc:161] P c81d9bc77ee54b4c8d04746984c5ed6e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5626e470e9c945789fb7f753593335a0 at {username='oryx', principal='oryx/127.8.88.188@KRBTEST.COM'} at 127.0.0.1:52192: session id = 5626e470e9c945789fb7f753593335a0-00000000000000000000000000000000
I20260928 13:08:22.945454 14666 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 13:08:22.945919 15297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 13:08:22.947180 15297 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: tablet copy: Starting download of 0 data blocks...
I20260928 13:08:22.947311 15297 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: tablet copy: Starting download of 1 WAL segments...
I20260928 13:08:22.948148 15297 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 13:08:22.949236 14666 tablet_copy_service.cc:342] P c81d9bc77ee54b4c8d04746984c5ed6e: Request end of tablet copy session 5626e470e9c945789fb7f753593335a0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.88.188@KRBTEST.COM'} at 127.0.0.1:52192
I20260928 13:08:22.949291 14666 tablet_copy_service.cc:433] P c81d9bc77ee54b4c8d04746984c5ed6e: ending tablet copy session 5626e470e9c945789fb7f753593335a0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5626e470e9c945789fb7f753593335a0
I20260928 13:08:22.951977 15297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:22.953533 15403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:22.953804 15297 server_base.cc:1034] running on GCE node
W20260928 13:08:22.957033 15406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:22.957893 15404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:22.958071 15297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:22.958420 15297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:22.959563 15297 hybrid_clock.cc:648] HybridClock initialized: now 1790600902959554 us; error 30 us; skew 500 ppm
Sep 28 13:08:22 dist-test-slave-p1lc krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600902, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:22.961285 15297 init.cc:377] Logged in from keytab as oryx/127.8.88.188@KRBTEST.COM (short username oryx)
I20260928 13:08:22.961596 15297 webserver.cc:460] Webserver started at http://127.8.88.188:41635/ using document root <none> and password file <none>
I20260928 13:08:22.961725 15297 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:22.961766 15297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:22.962651 15297 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:22.963310 15413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:22.963487 15297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:22.963546 15297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/wal
uuid: "5626e470e9c945789fb7f753593335a0"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "6c1d0f5098cac3cc306ed2529cce1865"
server_key_iv: "bf72d413f0cbf0cb3720cd74dead468a"
server_key_version: "encryptionkey@0"
I20260928 13:08:22.963619 15297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:22.979795 15297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:22.981805 15297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:22.985008 15297 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.188:45571
I20260928 13:08:22.985363 15297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/master-2/data/info.pb
I20260928 13:08:22.988991 15465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.188:45571 every 8 connection(s)
I20260928 13:08:22.989537 15466 sys_catalog.cc:263] Verifying existing consensus state
I20260928 13:08:22.989960 15466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: Bootstrap starting.
I20260928 13:08:22.992353 15466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:08:22.992583 15466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: Bootstrap complete.
I20260928 13:08:22.992803 15466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } }
I20260928 13:08:22.992915 15466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5626e470e9c945789fb7f753593335a0, State: Initialized, Role: LEARNER
I20260928 13:08:22.992970 15466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } }
I20260928 13:08:22.993194 15466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:22.993166 15469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } } }
I20260928 13:08:22.993371 15469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0 [sys.catalog]: This master's current role is: LEARNER
I20260928 13:08:22.994801 15480 catalog_manager.cc:1259] Loaded cluster ID: f1087f1fd83d400ab3e8de7dba38b4c0
I20260928 13:08:22.994891 15480 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: loading cluster ID for follower catalog manager: success
I20260928 13:08:22.995954 15480 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: acquiring CA information for follower catalog manager: success
I20260928 13:08:22.996248 15480 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5626e470e9c945789fb7f753593335a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 13:08:23.010600 15390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:23.010926 15390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:23.010970 15390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:23.013036 15390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:23.013123 15390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:23.013168 15390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:08:23.015856 15390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab
--principal=oryx/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:44799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=44747
--tserver_master_addrs=127.8.88.190:43479,127.8.88.189:37155,127.8.88.188:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:23.016199 15390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:23.016511 15390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:23.022269 15488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:23.022359 15486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:23.022341 15485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:23.022802 15390 server_base.cc:1034] running on GCE node
I20260928 13:08:23.023077 15390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:23.023379 15390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:23.024552 15390 hybrid_clock.cc:648] HybridClock initialized: now 1790600903024531 us; error 38 us; skew 500 ppm
Sep 28 13:08:23 dist-test-slave-p1lc krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600903, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:23.028024 15390 init.cc:377] Logged in from keytab as oryx/127.8.88.129@KRBTEST.COM (short username oryx)
I20260928 13:08:23.028679 15390 webserver.cc:460] Webserver started at http://127.8.88.129:44747/ using document root <none> and password file <none>
I20260928 13:08:23.029054 15390 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:23.029160 15390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:23.031279 15390 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:08:23.032020 15495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:23.032289 15390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:23.032393 15390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "0ed8b333e4314457a4364a01642d9da0"
format_stamp: "Formatted at 2026-09-28 13:08:21 on dist-test-slave-p1lc"
server_key: "80c716c09b98636cd8f3cd7a229a2b9d"
server_key_iv: "3ef69b4774454ba5b2772fd88e49ec7e"
server_key_version: "encryptionkey@0"
I20260928 13:08:23.032996 15390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:23.054294 15390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:23.056478 15390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:23.056607 15390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:23.056807 15390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:23.057166 15390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:23.057202 15390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:23.057246 15390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:23.057273 15390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:23.064569 15390 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:44799
I20260928 13:08:23.064699 15608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:44799 every 8 connection(s)
I20260928 13:08:23.065086 15390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
Sep 28 13:08:23 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600903, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
Sep 28 13:08:23 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600903, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.189@KRBTEST.COM
Sep 28 13:08:23 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600903, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.188@KRBTEST.COM
I20260928 13:08:23.075196 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 15390
I20260928 13:08:23.075356 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 14842
I20260928 13:08:23.078595 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.066473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:50673 (local address 127.8.88.189:37155)
0928 13:08:23.066523 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.066525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.066576 (+ 51us) server_negotiation.cc:407] Connection header received
0928 13:08:23.066903 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.066906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.066919 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.067010 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.067872 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.068384 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.069836 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.070135 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.070200 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.073539 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.073541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.073541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.073549 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.075660 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.077781 (+ 2121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.077783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.077783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.077838 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.078039 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.078040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.078040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.078179 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.078233 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.078413 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.078490 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 13:08:23.078601 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.066124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:45025 (local address 127.8.88.190:43479)
0928 13:08:23.066370 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.066372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.066384 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:08:23.066942 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.066944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.066956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.066989 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.068486 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.069057 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.070070 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.070288 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.070368 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.073583 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.073585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.073585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.073592 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.075457 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.077371 (+ 1914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.077373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.077373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.077438 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.077722 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.077723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.077724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.077892 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.077937 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.078347 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.078441 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260928 13:08:23.078734 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.065870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:45571 (local address 127.8.88.129:41749)
0928 13:08:23.066046 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.066093 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.066788 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.067160 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.067161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.067579 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:23.067773 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.067779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.069749 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.069751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.070552 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.070554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.071525 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.071526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.071599 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.072112 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:23.072116 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:23.073987 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:23.076795 (+ 2808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.076796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.076797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.077277 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.077501 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.077501 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.077502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.077633 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.078091 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:23.078091 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:23.078192 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.078304 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.078495 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":6,"thread_start_us":91,"threads_started":1}
I20260928 13:08:23.079073 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.066318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:41749 (local address 127.8.88.188:45571)
0928 13:08:23.066436 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.066439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.066447 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:08:23.067064 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.067065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.067077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.067143 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.068967 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.069668 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.071077 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.071435 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.071518 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.074082 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.074083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.074083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.074091 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.076725 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.077364 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.077365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.077365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.077430 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.077711 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.077712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.077713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.077987 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.078065 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.078869 (+ 804us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.078966 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260928 13:08:23.079070 15615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.066184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37155 (local address 127.8.88.129:50673)
0928 13:08:23.066520 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.066532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.066792 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.067075 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.067079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.067561 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:23.067773 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.067779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.068499 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.068500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.069746 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.069749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.070265 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.070265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.070426 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.071102 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:23.071111 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:23.073427 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:23.077428 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.077429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.077429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.077694 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.077912 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.077913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.077914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.077965 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.078268 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:23.078268 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:23.078307 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.078701 (+ 394us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.078978 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":11,"thread_start_us":57,"threads_started":1}
I20260928 13:08:23.079437 15611 heartbeater.cc:347] Connected to a master server at 127.8.88.189:37155
I20260928 13:08:23.079447 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.065869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:45025)
0928 13:08:23.066046 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.066094 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.066795 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.067077 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.067081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.067571 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:23.067771 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.067777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.069138 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.069139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.069948 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.069950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.070370 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.070370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.070433 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.071107 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:23.071113 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:23.073463 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:23.076769 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.076779 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.076782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.077277 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.077508 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.077508 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.077509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.077649 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.077997 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:23.078003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:23.078196 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.078787 (+ 591us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.079109 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":160,"spinlock_wait_cycles":384,"thread_start_us":108,"threads_started":1}
I20260928 13:08:23.079562 15611 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:23.079764 15611 heartbeater.cc:511] Master 127.8.88.189:37155 requested a full tablet report, sending...
I20260928 13:08:23.079835 15610 heartbeater.cc:347] Connected to a master server at 127.8.88.190:43479
I20260928 13:08:23.079882 15610 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:23.079954 15609 heartbeater.cc:347] Connected to a master server at 127.8.88.188:45571
I20260928 13:08:23.079954 15610 heartbeater.cc:511] Master 127.8.88.190:43479 requested a full tablet report, sending...
I20260928 13:08:23.080014 15609 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:23.080088 15609 heartbeater.cc:511] Master 127.8.88.188:45571 requested a full tablet report, sending...
I20260928 13:08:23.080318 15243 ts_manager.cc:194] Registered new tserver with Master: 0ed8b333e4314457a4364a01642d9da0 (127.8.88.129:44799)
I20260928 13:08:23.080399 15430 ts_manager.cc:194] Registered new tserver with Master: 0ed8b333e4314457a4364a01642d9da0 (127.8.88.129:44799)
I20260928 13:08:23.080705 14646 ts_manager.cc:194] Re-registered known tserver with Master: 0ed8b333e4314457a4364a01642d9da0 (127.8.88.129:44799)
I20260928 13:08:23.081346 14646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:45025
I20260928 13:08:23.081377 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:41485
--local_ip_for_outbound_sockets=127.8.88.130
--tserver_master_addrs=127.8.88.190:43479
--webserver_port=37163
--webserver_interface=127.8.88.130
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab
--principal=oryx/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.88.190:43479,127.8.88.189:37155
--tserver_master_addrs=127.8.88.190:43479,127.8.88.189:37155,127.8.88.188:45571 with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:23.171792 15617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:23.172034 15617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:23.172060 15617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:23.173408 15617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:23.173460 15617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:23.173491 15617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:08:23.175149 15617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab
--principal=oryx/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:41485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=37163
--tserver_master_addrs=127.8.88.190:43479,127.8.88.189:37155,127.8.88.188:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:23.175415 15617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:23.175676 15617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:23.181165 15623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:23.181339 15617 server_base.cc:1034] running on GCE node
W20260928 13:08:23.181372 15622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:23.181165 15625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:23.181739 15617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:23.182021 15617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:23.183208 15617 hybrid_clock.cc:648] HybridClock initialized: now 1790600903183190 us; error 34 us; skew 500 ppm
Sep 28 13:08:23 dist-test-slave-p1lc krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600903, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:23.186578 15617 init.cc:377] Logged in from keytab as oryx/127.8.88.130@KRBTEST.COM (short username oryx)
I20260928 13:08:23.187057 15617 webserver.cc:460] Webserver started at http://127.8.88.130:37163/ using document root <none> and password file <none>
I20260928 13:08:23.187309 15617 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:23.187389 15617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:23.188843 15617 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:08:23.189536 15632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:23.189724 15617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:08:23.189826 15617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "8756a6ddde784ad28cbdc9d1fb4183de"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "7977e7dae3ab3b541cefd31d7c20f31f"
server_key_iv: "91c7b4ea61a6c7bb0f2735f624c9edf7"
server_key_version: "encryptionkey@0"
I20260928 13:08:23.190191 15617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:23.208086 15617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:23.210826 15617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:23.211025 15617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:23.211295 15617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:23.211623 15617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:23.211674 15617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:23.211717 15617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:23.211755 15617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:23.219453 15617 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:41485
I20260928 13:08:23.219480 15745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:41485 every 8 connection(s)
I20260928 13:08:23.219921 15617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:08:23.226812 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 15617
I20260928 13:08:23.226997 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 14976
Sep 28 13:08:23 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600903, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.189@KRBTEST.COM
Sep 28 13:08:23 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600903, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.188@KRBTEST.COM
I20260928 13:08:23.232309 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.220694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37155 (local address 127.8.88.130:37015)
0928 13:08:23.220910 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.220983 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.221568 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.221891 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.221892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.222390 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:23.222603 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.222608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.223567 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.223568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.224689 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.224692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.225532 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.225533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.225631 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.226294 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:23.226306 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:23.228277 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:23.230282 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.230293 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.230296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.230630 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.230848 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.230849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.230850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.230975 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.231484 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:23.231492 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:23.231701 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.231824 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.232069 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":84,"thread_start_us":98,"threads_started":1}
I20260928 13:08:23.232733 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.220700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:37015 (local address 127.8.88.189:37155)
0928 13:08:23.220748 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.220750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.221076 (+ 326us) server_negotiation.cc:407] Connection header received
0928 13:08:23.221811 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.221812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.221823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.221878 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.222719 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.223480 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.225209 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.225430 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.225519 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.228392 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.228394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.228394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.228402 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.230180 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.230715 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.230716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.230717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.230784 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.231063 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.231065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.231065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.231374 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.231491 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.231845 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.231981 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:08:23.233083 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:45765
--local_ip_for_outbound_sockets=127.8.88.131
--tserver_master_addrs=127.8.88.190:43479
--webserver_port=43263
--webserver_interface=127.8.88.131
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab
--principal=oryx/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.88.190:43479,127.8.88.189:37155
--tserver_master_addrs=127.8.88.190:43479,127.8.88.189:37155,127.8.88.188:45571 with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:08:23.233364 15748 heartbeater.cc:347] Connected to a master server at 127.8.88.189:37155
I20260928 13:08:23.233516 15748 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:23.233554 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.221210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:54933 (local address 127.8.88.188:45571)
0928 13:08:23.221269 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.221272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.221281 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:08:23.221699 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.221701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.221713 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.221743 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.222720 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.223497 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.224637 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.225022 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.225095 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.230308 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.230310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.230311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.230319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.232077 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.232502 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.232504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.232504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.232560 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.232924 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.232925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.232925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.233086 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.233136 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.233360 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.233460 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 13:08:23.233604 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.221035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:45571 (local address 127.8.88.130:54933)
0928 13:08:23.221147 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.221157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.221569 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.221834 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.221837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.222400 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:23.222603 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.222608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.223585 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.223586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.224425 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.224428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.225138 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.225139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.225745 (+ 606us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.228632 (+ 2887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:23.228637 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:23.230180 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:23.232173 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.232174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.232175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.232415 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.232707 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.232708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.232708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.232792 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.233191 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:23.233192 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:23.233241 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.233313 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.233433 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260928 13:08:23.234014 15748 heartbeater.cc:511] Master 127.8.88.189:37155 requested a full tablet report, sending...
I20260928 13:08:23.234182 15746 heartbeater.cc:347] Connected to a master server at 127.8.88.188:45571
I20260928 13:08:23.234238 15746 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:23.234308 15746 heartbeater.cc:511] Master 127.8.88.188:45571 requested a full tablet report, sending...
I20260928 13:08:23.234519 15243 ts_manager.cc:194] Registered new tserver with Master: 8756a6ddde784ad28cbdc9d1fb4183de (127.8.88.130:41485)
I20260928 13:08:23.234920 15430 ts_manager.cc:194] Registered new tserver with Master: 8756a6ddde784ad28cbdc9d1fb4183de (127.8.88.130:41485)
Sep 28 13:08:23 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600903, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
I20260928 13:08:23.238759 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.220672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:41321 (local address 127.8.88.190:43479)
0928 13:08:23.220726 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.220728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.221042 (+ 314us) server_negotiation.cc:407] Connection header received
0928 13:08:23.221687 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.221688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.221702 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.221752 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.222717 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.223225 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.224742 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.225076 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.225146 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.235750 (+ 10604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.235753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.235753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.235762 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.237497 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.237857 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.237859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.237859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.237919 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.238139 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.238140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.238140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.238283 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.238363 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.238548 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.238624 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:08:23.239059 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.220693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:41321)
0928 13:08:23.220925 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.220956 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.221589 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.221834 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.221839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.222358 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:23.222602 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.222607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.223329 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.223330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.224540 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.224543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.225158 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.225159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.225762 (+ 603us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.226521 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:23.226526 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:23.235655 (+ 9129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:23.237589 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.237590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.237591 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.237779 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.238000 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.238001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.238001 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.238061 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.238353 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:23.238354 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:23.238402 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.238488 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.238616 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":29,"thread_start_us":103,"threads_started":1}
I20260928 13:08:23.239447 15747 heartbeater.cc:347] Connected to a master server at 127.8.88.190:43479
I20260928 13:08:23.239501 15747 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:23.239583 15747 heartbeater.cc:511] Master 127.8.88.190:43479 requested a full tablet report, sending...
I20260928 13:08:23.239861 14646 ts_manager.cc:194] Re-registered known tserver with Master: 8756a6ddde784ad28cbdc9d1fb4183de (127.8.88.130:41485)
I20260928 13:08:23.240291 14646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:41321
W20260928 13:08:23.324262 15753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:23.324666 15753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:23.324728 15753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:23.326062 15753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:23.326120 15753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:23.326150 15753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:08:23.327931 15753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab
--principal=oryx/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:45765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=43263
--tserver_master_addrs=127.8.88.190:43479,127.8.88.189:37155,127.8.88.188:45571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:23.328251 15753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:23.328552 15753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:23.334435 15759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:23.334550 15761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:23.334416 15758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:23.334784 15753 server_base.cc:1034] running on GCE node
I20260928 13:08:23.335045 15753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:23.335338 15753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:23.336527 15753 hybrid_clock.cc:648] HybridClock initialized: now 1790600903336512 us; error 34 us; skew 500 ppm
Sep 28 13:08:23 dist-test-slave-p1lc krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600903, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:23.339774 15753 init.cc:377] Logged in from keytab as oryx/127.8.88.131@KRBTEST.COM (short username oryx)
I20260928 13:08:23.340277 15753 webserver.cc:460] Webserver started at http://127.8.88.131:43263/ using document root <none> and password file <none>
I20260928 13:08:23.340535 15753 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:23.340622 15753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:23.342177 15753 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:23.342928 15768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:23.343201 15753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:23.343317 15753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "503b2cb44b85450e92f6ca9d3b086f43"
format_stamp: "Formatted at 2026-09-28 13:08:22 on dist-test-slave-p1lc"
server_key: "cf8b9c8ca02bb8a4dea6cef685b0fcdf"
server_key_iv: "ad26532a5c72e3087e333ec2d67c891c"
server_key_version: "encryptionkey@0"
I20260928 13:08:23.343743 15753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:23.380648 15753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:23.383747 15753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:23.383947 15753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:23.384217 15753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:23.384585 15753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:23.384640 15753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:23.384683 15753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:23.384720 15753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:23.391862 15753 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:45765
I20260928 13:08:23.391932 15881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:45765 every 8 connection(s)
I20260928 13:08:23.392320 15753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:08:23.399935 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 15753
Sep 28 13:08:23 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600903, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.189@KRBTEST.COM
Sep 28 13:08:23 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600903, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.188@KRBTEST.COM
Sep 28 13:08:23 dist-test-slave-p1lc krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600903, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
I20260928 13:08:23.411372 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.393193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:36473)
0928 13:08:23.393365 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.393410 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.394668 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.394930 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.394934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.395365 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:23.395604 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.395611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.397217 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.397218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.397961 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.397964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.398379 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.398380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.398543 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.399364 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:23.399369 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:23.403367 (+ 3998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:23.406268 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.406282 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.406285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.406648 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.409581 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.409583 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.409583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.409754 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.410293 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:23.410303 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:23.410511 (+ 208us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.410644 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.410899 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":25,"thread_start_us":90,"threads_started":1}
I20260928 13:08:23.411468 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.393184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:58703 (local address 127.8.88.189:37155)
0928 13:08:23.393245 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.393248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.393492 (+ 244us) server_negotiation.cc:407] Connection header received
0928 13:08:23.394799 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.394801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.394814 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.394848 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.395719 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.396278 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.398075 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.398375 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.398418 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.402471 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.402473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.402473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.402483 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.408955 (+ 6472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.409385 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.409386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.409387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.409463 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.409836 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.409838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.409838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.410203 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.410266 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.410789 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.411354 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:08:23.411623 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.393252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:36473 (local address 127.8.88.190:43479)
0928 13:08:23.393302 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.393304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.393518 (+ 214us) server_negotiation.cc:407] Connection header received
0928 13:08:23.394788 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.394790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.394804 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.394851 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.396411 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.397135 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.398047 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.398295 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.398373 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.403458 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.403459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.403460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.403480 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.406165 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.409398 (+ 3233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.409399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.409399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.409497 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.409888 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.409889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.409890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.410071 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.410226 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.411429 (+ 1203us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.411522 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:08:23.411924 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.401156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.8.88.190:43479)
0928 13:08:23.401309 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.401311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.401320 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:08:23.401348 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.401348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.401358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.401417 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.402322 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.403044 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.404474 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.404827 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.404982 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.406758 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.406760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.406760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.406769 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.408932 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.410879 (+ 1947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.410880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.410881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.410948 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.411172 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.411173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.411173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.411315 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.411415 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.411758 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.411837 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":103,"threads_started":1}
I20260928 13:08:23.411378 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.393193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37155 (local address 127.8.88.131:58703)
0928 13:08:23.393365 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.393410 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.394660 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.394931 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.394936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.395371 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:23.395605 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.395612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.396419 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.396420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.397965 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.397968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.398504 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.398505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.398589 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.399258 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:23.399270 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:23.402342 (+ 3072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:23.409050 (+ 6708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.409051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.409052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.409270 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.409537 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.409538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.409539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.409754 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.410293 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:23.410303 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:23.410597 (+ 294us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.410675 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.411176 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":413,"thread_start_us":100,"threads_started":1}
I20260928 13:08:23.412066 15883 heartbeater.cc:347] Connected to a master server at 127.8.88.190:43479
I20260928 13:08:23.412216 15883 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:23.412568 15883 heartbeater.cc:511] Master 127.8.88.190:43479 requested a full tablet report, sending...
I20260928 13:08:23.413120 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.393576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:45571 (local address 127.8.88.131:46125)
0928 13:08:23.393699 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.393710 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.394671 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.394931 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.394936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.395367 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:23.395605 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.395612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.396419 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.396420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.397965 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.397968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.398505 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.398505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.398569 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.399294 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:23.399298 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:23.403261 (+ 3963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:23.410257 (+ 6996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.410258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.410258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.410576 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.411033 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:23.411034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:23.411034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:23.411113 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:23.412530 (+ 1417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:23.412531 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:23.412578 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.412649 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.412765 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":78,"thread_start_us":66,"threads_started":1}
I20260928 13:08:23.413434 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.393669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:46125 (local address 127.8.88.188:45571)
0928 13:08:23.393723 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.393726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.393755 (+ 29us) server_negotiation.cc:407] Connection header received
0928 13:08:23.394799 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.394800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.394814 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.394848 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.395731 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.396315 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.398090 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.398393 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.398431 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.403370 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.403373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.403373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.403383 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.410158 (+ 6775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.410883 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.410884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.410884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.410955 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.412250 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.412251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.412251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.412420 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.412488 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.413231 (+ 743us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.413339 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 13:08:23.414330 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.403806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.8.88.188:45571)
0928 13:08:23.403910 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.403913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.404152 (+ 239us) server_negotiation.cc:407] Connection header received
0928 13:08:23.404178 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.404179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.404193 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.404260 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.404994 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.405805 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.406930 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.407254 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.407306 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.408854 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.408855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.408856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.408894 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.413040 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.413463 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.413464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.413465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.413532 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.413726 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.413727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.413728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.413901 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.414007 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.414116 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.414206 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":58,"threads_started":1}
I20260928 13:08:23.414455 15882 heartbeater.cc:347] Connected to a master server at 127.8.88.188:45571
I20260928 13:08:23.414505 15882 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:23.414598 15882 heartbeater.cc:511] Master 127.8.88.188:45571 requested a full tablet report, sending...
I20260928 13:08:23.414754 14646 ts_manager.cc:194] Re-registered known tserver with Master: 503b2cb44b85450e92f6ca9d3b086f43 (127.8.88.131:45765)
I20260928 13:08:23.414896 15429 ts_manager.cc:194] Registered new tserver with Master: 503b2cb44b85450e92f6ca9d3b086f43 (127.8.88.131:45765)
I20260928 13:08:23.415256 14646 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:36473
I20260928 13:08:23.416028 15884 heartbeater.cc:347] Connected to a master server at 127.8.88.189:37155
I20260928 13:08:23.416085 15884 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:23.416177 15884 heartbeater.cc:511] Master 127.8.88.189:37155 requested a full tablet report, sending...
I20260928 13:08:23.416507 15243 ts_manager.cc:194] Registered new tserver with Master: 503b2cb44b85450e92f6ca9d3b086f43 (127.8.88.131:45765)
I20260928 13:08:23.415956 14646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:08:23.418565 14646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:23.419262 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.403419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.8.88.189:37155)
0928 13:08:23.405128 (+ 1709us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.405132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.405140 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:08:23.405165 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.405166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.405180 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.405279 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.406153 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.407250 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.408558 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.408922 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.409243 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.410609 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.410610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.410611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.410618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.417728 (+ 7110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.418193 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.418194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.418195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.418268 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.418509 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.418510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.418511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.418648 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.418875 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.419055 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.419172 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1684,"thread_start_us":61,"threads_started":1}
I20260928 13:08:23.427350 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.423450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:44799 (local address 127.0.0.1:58330)
0928 13:08:23.423859 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.423872 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.423934 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.424138 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.424140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.424142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:23.424202 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.424204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.425277 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.425279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.426333 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.426337 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.427082 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.427083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.427146 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.427151 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.427208 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.427267 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":48,"threads_started":1}
I20260928 13:08:23.427614 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.423628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.8.88.129:44799)
0928 13:08:23.423824 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.423831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.423937 (+ 106us) server_negotiation.cc:407] Connection header received
0928 13:08:23.424006 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.424008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.424023 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.424125 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:23.424264 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.425193 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.426450 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.426975 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.427080 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.427170 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.427228 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.427265 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260928 13:08:23.428488 15905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.423513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:41485 (local address 127.0.0.1:47300)
0928 13:08:23.424012 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.424020 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.424062 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.424411 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.424412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.424414 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:23.424475 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.424478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.425770 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.425771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.427081 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.427083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.428198 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.428199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.428277 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.428281 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.428327 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.428376 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":62,"thread_start_us":65,"threads_started":1}
I20260928 13:08:23.428387 15543 tablet_service.cc:1467] Processing CreateTablet for tablet 64650885ab6f47bd92ff2d0a1f70811e (DEFAULT_TABLE table=test-table [id=29cf6f1c458440e897d4d830615c7165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:23.428821 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.423705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.8.88.130:41485)
0928 13:08:23.423862 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.423867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.424066 (+ 199us) server_negotiation.cc:407] Connection header received
0928 13:08:23.424299 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.424301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.424318 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.424395 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:23.424579 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.425668 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.427187 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.427969 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.428045 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.428646 (+ 601us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.428677 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.428724 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260928 13:08:23.428841 15543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64650885ab6f47bd92ff2d0a1f70811e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:23.429600 15680 tablet_service.cc:1467] Processing CreateTablet for tablet 64650885ab6f47bd92ff2d0a1f70811e (DEFAULT_TABLE table=test-table [id=29cf6f1c458440e897d4d830615c7165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:23.429976 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.423455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:45765 (local address 127.0.0.1:42452)
0928 13:08:23.423868 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.423878 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.423933 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.424395 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.424396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.424397 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:23.424432 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.424434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.425770 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.425771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.427080 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.427083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.429699 (+ 2616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.429700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.429768 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.429773 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.429829 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.429879 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":57,"thread_start_us":52,"threads_started":1}
I20260928 13:08:23.430006 15680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64650885ab6f47bd92ff2d0a1f70811e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:23.430361 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.423631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.8.88.131:45765)
0928 13:08:23.424182 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.424186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.424195 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:08:23.424231 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.424234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.424251 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.424326 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:23.424499 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.425663 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.427187 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.429467 (+ 2280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.429567 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.429657 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.430221 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.430280 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":81,"threads_started":1}
I20260928 13:08:23.431135 15816 tablet_service.cc:1467] Processing CreateTablet for tablet 64650885ab6f47bd92ff2d0a1f70811e (DEFAULT_TABLE table=test-table [id=29cf6f1c458440e897d4d830615c7165]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:23.431501 15816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64650885ab6f47bd92ff2d0a1f70811e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:23.433082 15910 tablet_bootstrap.cc:492] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0: Bootstrap starting.
I20260928 13:08:23.433462 15911 tablet_bootstrap.cc:492] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de: Bootstrap starting.
I20260928 13:08:23.433704 15910 tablet_bootstrap.cc:654] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:23.433992 15910 log.cc:824] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:23.434139 15911 tablet_bootstrap.cc:654] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:23.434453 15911 log.cc:824] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:23.435266 15911 tablet_bootstrap.cc:492] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de: No bootstrap required, opened a new log
I20260928 13:08:23.435328 15911 ts_tablet_manager.cc:1397] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:23.435668 15914 tablet_bootstrap.cc:492] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43: Bootstrap starting.
I20260928 13:08:23.436352 15914 tablet_bootstrap.cc:654] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:23.436614 15910 tablet_bootstrap.cc:492] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0: No bootstrap required, opened a new log
I20260928 13:08:23.436673 15914 log.cc:824] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:23.436693 15910 ts_tablet_manager.cc:1397] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 13:08:23.437518 15914 tablet_bootstrap.cc:492] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43: No bootstrap required, opened a new log
I20260928 13:08:23.437631 15914 ts_tablet_manager.cc:1397] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 13:08:23.437975 15911 raft_consensus.cc:348] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503b2cb44b85450e92f6ca9d3b086f43" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 45765 } } peers { permanent_uuid: "0ed8b333e4314457a4364a01642d9da0" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44799 } } peers { permanent_uuid: "8756a6ddde784ad28cbdc9d1fb4183de" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 41485 } }
I20260928 13:08:23.438138 15911 raft_consensus.cc:374] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:23.438181 15911 raft_consensus.cc:729] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8756a6ddde784ad28cbdc9d1fb4183de, State: Initialized, Role: FOLLOWER
I20260928 13:08:23.438295 15911 consensus_queue.cc:260] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503b2cb44b85450e92f6ca9d3b086f43" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 45765 } } peers { permanent_uuid: "0ed8b333e4314457a4364a01642d9da0" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44799 } } peers { permanent_uuid: "8756a6ddde784ad28cbdc9d1fb4183de" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 41485 } }
I20260928 13:08:23.438570 15911 ts_tablet_manager.cc:1428] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 13:08:23.438653 15747 heartbeater.cc:503] Master 127.8.88.190:43479 was elected leader, sending a full tablet report...
I20260928 13:08:23.439517 15910 raft_consensus.cc:348] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503b2cb44b85450e92f6ca9d3b086f43" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 45765 } } peers { permanent_uuid: "0ed8b333e4314457a4364a01642d9da0" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44799 } } peers { permanent_uuid: "8756a6ddde784ad28cbdc9d1fb4183de" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 41485 } }
I20260928 13:08:23.439714 15910 raft_consensus.cc:374] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:23.439761 15910 raft_consensus.cc:729] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed8b333e4314457a4364a01642d9da0, State: Initialized, Role: FOLLOWER
I20260928 13:08:23.439879 15910 consensus_queue.cc:260] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503b2cb44b85450e92f6ca9d3b086f43" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 45765 } } peers { permanent_uuid: "0ed8b333e4314457a4364a01642d9da0" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44799 } } peers { permanent_uuid: "8756a6ddde784ad28cbdc9d1fb4183de" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 41485 } }
I20260928 13:08:23.440182 15910 ts_tablet_manager.cc:1428] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 13:08:23.440330 15610 heartbeater.cc:503] Master 127.8.88.190:43479 was elected leader, sending a full tablet report...
I20260928 13:08:23.441550 15914 raft_consensus.cc:348] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503b2cb44b85450e92f6ca9d3b086f43" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 45765 } } peers { permanent_uuid: "0ed8b333e4314457a4364a01642d9da0" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44799 } } peers { permanent_uuid: "8756a6ddde784ad28cbdc9d1fb4183de" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 41485 } }
I20260928 13:08:23.441711 15914 raft_consensus.cc:374] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:23.441733 15914 raft_consensus.cc:729] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503b2cb44b85450e92f6ca9d3b086f43, State: Initialized, Role: FOLLOWER
I20260928 13:08:23.441852 15914 consensus_queue.cc:260] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503b2cb44b85450e92f6ca9d3b086f43" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 45765 } } peers { permanent_uuid: "0ed8b333e4314457a4364a01642d9da0" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44799 } } peers { permanent_uuid: "8756a6ddde784ad28cbdc9d1fb4183de" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 41485 } }
I20260928 13:08:23.442293 15883 heartbeater.cc:503] Master 127.8.88.190:43479 was elected leader, sending a full tablet report...
I20260928 13:08:23.442339 15914 ts_tablet_manager.cc:1428] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43: Time spent starting tablet: real 0.005s user 0.003s sys 0.001s
I20260928 13:08:23.450806 15918 raft_consensus.cc:482] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:23.450953 15918 raft_consensus.cc:504] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503b2cb44b85450e92f6ca9d3b086f43" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 45765 } } peers { permanent_uuid: "0ed8b333e4314457a4364a01642d9da0" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44799 } } peers { permanent_uuid: "8756a6ddde784ad28cbdc9d1fb4183de" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 41485 } }
I20260928 13:08:23.451440 15918 leader_election.cc:288] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed8b333e4314457a4364a01642d9da0 (127.8.88.129:44799), 8756a6ddde784ad28cbdc9d1fb4183de (127.8.88.130:41485)
I20260928 13:08:23.454458 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.451546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:44799 (local address 127.8.88.131:38897)
0928 13:08:23.451648 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.451663 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.451711 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.451856 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.451857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.451858 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:23.451942 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.451944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.452659 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.452661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.453546 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.453557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.454184 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.454185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.454265 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.454268 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.454334 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.454378 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 13:08:23.454481 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.451613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:38897 (local address 127.8.88.129:44799)
0928 13:08:23.451656 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.451659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.451720 (+ 61us) server_negotiation.cc:407] Connection header received
0928 13:08:23.451780 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.451781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.451805 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.451855 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:23.452109 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.452578 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.453668 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.454087 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.454184 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.454195 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.454341 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.454378 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:08:23.454995 15563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64650885ab6f47bd92ff2d0a1f70811e" candidate_uuid: "503b2cb44b85450e92f6ca9d3b086f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed8b333e4314457a4364a01642d9da0" is_pre_election: true
I20260928 13:08:23.455199 15563 raft_consensus.cc:2454] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 503b2cb44b85450e92f6ca9d3b086f43 in term 0.
I20260928 13:08:23.455440 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.451588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:41485 (local address 127.8.88.131:41351)
0928 13:08:23.451671 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:23.451680 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:23.451721 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:23.451913 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:23.451914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:23.451915 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:23.451982 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.451984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.452615 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.452616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.454114 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:23.454117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.455174 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.455175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:23.455252 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.455255 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:08:23.455321 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 13:08:23.455366 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 13:08:23.455427 15770 leader_election.cc:302] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ed8b333e4314457a4364a01642d9da0, 503b2cb44b85450e92f6ca9d3b086f43; no voters:
I20260928 13:08:23.455606 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.451750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:41351 (local address 127.8.88.130:41485)
0928 13:08:23.451795 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.451797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.451805 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:08:23.451830 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.451831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.451846 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.451910 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:23.452049 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.452530 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.454307 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.455052 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.455147 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.455158 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.455476 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.455512 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 13:08:23.455626 15918 raft_consensus.cc:2781] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:23.455698 15918 raft_consensus.cc:482] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:23.455732 15918 raft_consensus.cc:3037] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:23.455881 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64650885ab6f47bd92ff2d0a1f70811e" candidate_uuid: "503b2cb44b85450e92f6ca9d3b086f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8756a6ddde784ad28cbdc9d1fb4183de" is_pre_election: true
I20260928 13:08:23.456043 15700 raft_consensus.cc:2454] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 503b2cb44b85450e92f6ca9d3b086f43 in term 0.
W20260928 13:08:23.470887 15749 tablet.cc:2369] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:08:23.471048 15918 raft_consensus.cc:504] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503b2cb44b85450e92f6ca9d3b086f43" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 45765 } } peers { permanent_uuid: "0ed8b333e4314457a4364a01642d9da0" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44799 } } peers { permanent_uuid: "8756a6ddde784ad28cbdc9d1fb4183de" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 41485 } }
I20260928 13:08:23.471256 15918 leader_election.cc:288] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [CANDIDATE]: Term 1 election: Requested vote from peers 0ed8b333e4314457a4364a01642d9da0 (127.8.88.129:44799), 8756a6ddde784ad28cbdc9d1fb4183de (127.8.88.130:41485)
I20260928 13:08:23.471504 15700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64650885ab6f47bd92ff2d0a1f70811e" candidate_uuid: "503b2cb44b85450e92f6ca9d3b086f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8756a6ddde784ad28cbdc9d1fb4183de"
I20260928 13:08:23.471500 15563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64650885ab6f47bd92ff2d0a1f70811e" candidate_uuid: "503b2cb44b85450e92f6ca9d3b086f43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed8b333e4314457a4364a01642d9da0"
I20260928 13:08:23.471627 15700 raft_consensus.cc:3037] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:23.471628 15563 raft_consensus.cc:3037] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:23.472383 15700 raft_consensus.cc:2454] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 503b2cb44b85450e92f6ca9d3b086f43 in term 1.
I20260928 13:08:23.472644 15769 leader_election.cc:302] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 503b2cb44b85450e92f6ca9d3b086f43, 8756a6ddde784ad28cbdc9d1fb4183de; no voters:
I20260928 13:08:23.472733 15563 raft_consensus.cc:2454] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 503b2cb44b85450e92f6ca9d3b086f43 in term 1.
I20260928 13:08:23.472766 15918 raft_consensus.cc:2781] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:23.472945 15918 raft_consensus.cc:686] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [term 1 LEADER]: Becoming Leader. State: Replica: 503b2cb44b85450e92f6ca9d3b086f43, State: Running, Role: LEADER
I20260928 13:08:23.473059 15918 consensus_queue.cc:237] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503b2cb44b85450e92f6ca9d3b086f43" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 45765 } } peers { permanent_uuid: "0ed8b333e4314457a4364a01642d9da0" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44799 } } peers { permanent_uuid: "8756a6ddde784ad28cbdc9d1fb4183de" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 41485 } }
I20260928 13:08:23.473975 14646 catalog_manager.cc:5495] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 503b2cb44b85450e92f6ca9d3b086f43 (127.8.88.131). New cstate: current_term: 1 leader_uuid: "503b2cb44b85450e92f6ca9d3b086f43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503b2cb44b85450e92f6ca9d3b086f43" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 45765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ed8b333e4314457a4364a01642d9da0" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8756a6ddde784ad28cbdc9d1fb4183de" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 41485 } health_report { overall_health: UNKNOWN } } }
I20260928 13:08:23.487447 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.484202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.8.88.131:45765)
0928 13:08:23.484262 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.484264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.484273 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:08:23.484349 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.484350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.484371 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.484452 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:23.484677 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.485257 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.486041 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.486285 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.486343 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.487097 (+ 754us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:23.487206 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:23.487278 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.487310 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.487331 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 13:08:23.489686 15700 raft_consensus.cc:1261] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de [term 1 FOLLOWER]: Refusing update from remote peer 503b2cb44b85450e92f6ca9d3b086f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:23.489903 15563 raft_consensus.cc:1261] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0 [term 1 FOLLOWER]: Refusing update from remote peer 503b2cb44b85450e92f6ca9d3b086f43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:23.490100 15919 consensus_queue.cc:1035] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8756a6ddde784ad28cbdc9d1fb4183de" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 41485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:23.490309 15918 consensus_queue.cc:1035] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed8b333e4314457a4364a01642d9da0" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:23.494109 15925 mvcc.cc:205] Tried to move back new op lower bound from 7334301300690759680 to 7334301300626419712. Current Snapshot: MvccSnapshot[applied={T|T < 7334301300690759680 or (T in {7334301300690759680})}]
I20260928 13:08:23.497634 14646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:23.497728 14646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:23.499197 14646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c81d9bc77ee54b4c8d04746984c5ed6e: Sending DeleteTablet for 3 replicas of tablet 64650885ab6f47bd92ff2d0a1f70811e
I20260928 13:08:23.499608 15680 tablet_service.cc:1514] Processing DeleteTablet for tablet 64650885ab6f47bd92ff2d0a1f70811e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:23 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:47300
I20260928 13:08:23.499872 15543 tablet_service.cc:1514] Processing DeleteTablet for tablet 64650885ab6f47bd92ff2d0a1f70811e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:23 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:58330
I20260928 13:08:23.500133 15933 tablet_replica.cc:330] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0: stopping tablet replica
I20260928 13:08:23.500252 15933 raft_consensus.cc:2229] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:23.500239 15816 tablet_service.cc:1514] Processing DeleteTablet for tablet 64650885ab6f47bd92ff2d0a1f70811e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:23 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:42452
I20260928 13:08:23.500365 15933 raft_consensus.cc:2258] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:23.500442 15935 tablet_replica.cc:330] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43: stopping tablet replica
I20260928 13:08:23.500530 15935 raft_consensus.cc:2229] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:08:23.500653 15935 raft_consensus.cc:2258] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:23.500902 15933 ts_tablet_manager.cc:1905] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:23.501242 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 14614
I20260928 13:08:23.501288 15935 ts_tablet_manager.cc:1905] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:23.506692 15932 tablet_replica.cc:330] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de: stopping tablet replica
I20260928 13:08:23.506882 15932 raft_consensus.cc:2229] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:23.507035 15932 raft_consensus.cc:2258] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:23.507627 15932 ts_tablet_manager.cc:1905] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:23.514885 15897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.500290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.8.88.188:45571)
0928 13:08:23.500336 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.500339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.505377 (+ 5038us) server_negotiation.cc:407] Connection header received
0928 13:08:23.505508 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.505509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.505526 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.505558 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.506608 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.507462 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.508636 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.509006 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.509074 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.511514 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.511516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.511516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.511524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.513401 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.513856 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.513857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.513858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.513914 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.514130 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.514130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.514131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.514324 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.514407 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.514681 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.514776 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:08:23.521335 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.511080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.8.88.189:37155)
0928 13:08:23.511148 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.511150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.511219 (+ 69us) server_negotiation.cc:407] Connection header received
0928 13:08:23.511286 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.511288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.511303 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.511385 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.512174 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.513006 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.514246 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.514562 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.514600 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.516934 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.516937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.516937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.516946 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.519157 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.519910 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.519912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.519912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.519974 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.520197 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.520199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.520199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.520361 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.520462 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.521139 (+ 677us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.521239 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 13:08:23.521525 15616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:23.510975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.8.88.188:45571)
0928 13:08:23.511036 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:23.511039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:23.511382 (+ 343us) server_negotiation.cc:407] Connection header received
0928 13:08:23.511478 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:23.511479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:23.511504 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:23.511536 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:23.512187 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.512975 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.514245 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:23.514553 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:23.514598 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:23.517188 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:23.517189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:23.517190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:23.517198 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:23.519202 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.519730 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.519731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.519732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.519804 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:23.520037 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:23.520038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:23.520039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:23.520212 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:23.520364 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:23.520447 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 13:08:23.521397 (+ 950us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 13:08:23.567597 15933 ts_tablet_manager.cc:1918] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:23.567729 15933 log.cc:1196] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-0/wal/wals/64650885ab6f47bd92ff2d0a1f70811e
I20260928 13:08:23.567729 15935 ts_tablet_manager.cc:1918] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:23.567857 15935 log.cc:1196] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/64650885ab6f47bd92ff2d0a1f70811e
I20260928 13:08:23.568029 15933 ts_tablet_manager.cc:1939] T 64650885ab6f47bd92ff2d0a1f70811e P 0ed8b333e4314457a4364a01642d9da0: Deleting consensus metadata
I20260928 13:08:23.568133 15935 ts_tablet_manager.cc:1939] T 64650885ab6f47bd92ff2d0a1f70811e P 503b2cb44b85450e92f6ca9d3b086f43: Deleting consensus metadata
W20260928 13:08:23.585603 15770 connection.cc:441] server connection from 127.0.0.1:42452 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42452 (request call id 1) could send its response
W20260928 13:08:23.587267 15498 connection.cc:441] server connection from 127.0.0.1:58330 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58330 (request call id 1) could send its response
I20260928 13:08:23.619386 15932 ts_tablet_manager.cc:1918] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:23.619542 15932 log.cc:1196] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/64650885ab6f47bd92ff2d0a1f70811e
I20260928 13:08:23.619879 15932 ts_tablet_manager.cc:1939] T 64650885ab6f47bd92ff2d0a1f70811e P 8756a6ddde784ad28cbdc9d1fb4183de: Deleting consensus metadata
W20260928 13:08:23.620416 15634 connection.cc:441] server connection from 127.0.0.1:47300 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47300 (request call id 1) could send its response
I20260928 13:08:24.494884 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:24.494514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:38745)
0928 13:08:24.494707 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:24.494822 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":78,"threads_started":1}
W20260928 13:08:24.495308 15610 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:43479 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
I20260928 13:08:24.495942 15944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:24.495659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:39213)
0928 13:08:24.495830 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:24.495894 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":55,"threads_started":1}
W20260928 13:08:24.496248 15883 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:43479 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
I20260928 13:08:24.496522 15945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:24.496246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:44871)
0928 13:08:24.496349 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:24.496476 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":50,"threads_started":1}
W20260928 13:08:24.496722 15747 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:43479 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
I20260928 13:08:24.999748 15946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c81d9bc77ee54b4c8d04746984c5ed6e)
I20260928 13:08:24.999879 15946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } }
I20260928 13:08:25.000087 15946 leader_election.cc:288] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479)
I20260928 13:08:25.000418 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:25.000226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:52748)
0928 13:08:25.000325 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:25.000375 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":59,"threads_started":1}
W20260928 13:08:25.000553 15230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
I20260928 13:08:25.001199 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:25.001091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:52764)
0928 13:08:25.001143 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:25.001170 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 13:08:25.001286 15230 leader_election.cc:334] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479): Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
I20260928 13:08:25.001340 15230 leader_election.cc:302] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 81409d6217b64c2794dca75da4b7e534; no voters: c81d9bc77ee54b4c8d04746984c5ed6e
I20260928 13:08:25.001434 15946 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:08:25.495982 15949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:25.495718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:54603)
0928 13:08:25.495855 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:25.495920 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260928 13:08:25.496814 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:25.496583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:44033)
0928 13:08:25.496716 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:25.496753 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260928 13:08:25.497408 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:25.497015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:55497)
0928 13:08:25.497326 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:25.497363 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":69,"threads_started":1}
I20260928 13:08:26.496918 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:26.496629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:50739)
0928 13:08:26.496758 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:26.496808 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260928 13:08:26.497802 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:26.497563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:43839)
0928 13:08:26.497710 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:26.497751 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260928 13:08:26.497959 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:26.497789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:50811)
0928 13:08:26.497875 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:26.497911 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":48,"threads_started":1}
I20260928 13:08:27.062597 15955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c81d9bc77ee54b4c8d04746984c5ed6e)
I20260928 13:08:27.062697 15955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } }
I20260928 13:08:27.062915 15955 leader_election.cc:288] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479)
I20260928 13:08:27.063272 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:27.063055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:53028)
0928 13:08:27.063189 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:27.063232 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260928 13:08:27.063683 15956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:27.063594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:53038)
0928 13:08:27.063639 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:27.063665 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 13:08:27.063757 15230 leader_election.cc:334] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479): Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
I20260928 13:08:27.063804 15230 leader_election.cc:302] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 81409d6217b64c2794dca75da4b7e534; no voters: c81d9bc77ee54b4c8d04746984c5ed6e
I20260928 13:08:27.063880 15955 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:08:27.497658 15958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:27.497425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:43389)
0928 13:08:27.497548 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:27.497594 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260928 13:08:27.498579 15959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:27.498377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:57519)
0928 13:08:27.498497 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:27.498532 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260928 13:08:27.498621 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:27.498409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:35965)
0928 13:08:27.498538 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:27.498579 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260928 13:08:28.498386 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:28.498162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:37013)
0928 13:08:28.498293 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:28.498339 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260928 13:08:28.499289 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:28.499081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:59715)
0928 13:08:28.499201 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:28.499237 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260928 13:08:28.499436 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:28.499094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:51923)
0928 13:08:28.499350 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:28.499387 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":57,"threads_started":1}
I20260928 13:08:28.966586 15964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c81d9bc77ee54b4c8d04746984c5ed6e)
I20260928 13:08:28.966684 15964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } }
I20260928 13:08:28.966832 15964 leader_election.cc:288] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479)
I20260928 13:08:28.967227 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:28.966983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:53224)
0928 13:08:28.967142 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:28.967188 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260928 13:08:28.967758 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:28.967667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:53234)
0928 13:08:28.967713 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:28.967740 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 13:08:28.967869 15230 leader_election.cc:334] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479): Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
I20260928 13:08:28.967948 15230 leader_election.cc:302] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 81409d6217b64c2794dca75da4b7e534; no voters: c81d9bc77ee54b4c8d04746984c5ed6e
I20260928 13:08:28.968037 15964 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:08:29.499146 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:29.498899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:35167)
0928 13:08:29.499039 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:29.499089 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260928 13:08:29.499996 15968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:29.499799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:37941)
0928 13:08:29.499917 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:29.499950 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260928 13:08:29.500260 15969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:29.499799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:36817)
0928 13:08:29.500144 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:29.500213 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":58,"threads_started":1}
I20260928 13:08:30.499997 15970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:30.499672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:38017)
0928 13:08:30.499865 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:30.499935 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":93,"threads_started":1}
I20260928 13:08:30.500676 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:30.500441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:45439)
0928 13:08:30.500586 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:30.500628 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":52,"threads_started":1}
I20260928 13:08:30.500905 15972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:30.500662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:34101)
0928 13:08:30.500804 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:30.500837 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":45,"threads_started":1}
I20260928 13:08:31.500860 15973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:31.500588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:52965)
0928 13:08:31.500722 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:31.500767 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260928 13:08:31.501451 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:31.501230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:35205)
0928 13:08:31.501364 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:31.501400 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":53,"threads_started":1}
I20260928 13:08:31.501537 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:31.501328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:42899)
0928 13:08:31.501454 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:31.501491 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":48,"threads_started":1}
I20260928 13:08:31.725175 15976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c81d9bc77ee54b4c8d04746984c5ed6e)
I20260928 13:08:31.725328 15976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } }
I20260928 13:08:31.725538 15976 leader_election.cc:288] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479)
I20260928 13:08:31.725894 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:31.725694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:54022)
0928 13:08:31.725809 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:31.725855 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
W20260928 13:08:31.726017 15230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260928 13:08:31.726495 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:31.726359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:54026)
0928 13:08:31.726439 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:31.726468 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 13:08:31.726600 15230 leader_election.cc:334] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479): Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
I20260928 13:08:31.726646 15230 leader_election.cc:302] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 81409d6217b64c2794dca75da4b7e534; no voters: c81d9bc77ee54b4c8d04746984c5ed6e
I20260928 13:08:31.726729 15976 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:08:32.502085 15979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:32.501724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:49893)
0928 13:08:32.501979 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:32.502024 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":64,"threads_started":1}
I20260928 13:08:32.502202 15981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:32.501965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:57997)
0928 13:08:32.502114 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:32.502151 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260928 13:08:32.502461 15980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:32.501978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:45185)
0928 13:08:32.502380 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:32.502416 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":89,"threads_started":1}
I20260928 13:08:33.503119 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:33.502634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:37265)
0928 13:08:33.503008 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:33.503054 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":79,"threads_started":1}
I20260928 13:08:33.503145 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:33.502915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:40583)
0928 13:08:33.503059 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:33.503094 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260928 13:08:33.503314 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:33.502656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:53535)
0928 13:08:33.503239 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:33.503273 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":61,"threads_started":1}
I20260928 13:08:33.668365 15985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c81d9bc77ee54b4c8d04746984c5ed6e)
I20260928 13:08:33.668509 15985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } }
I20260928 13:08:33.668955 15985 leader_election.cc:288] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479)
I20260928 13:08:33.669345 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:33.669065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:54188)
0928 13:08:33.669270 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:33.669309 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20260928 13:08:33.669848 15986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:33.669718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:54200)
0928 13:08:33.669790 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:33.669827 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 13:08:33.669929 15230 leader_election.cc:334] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479): Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
I20260928 13:08:33.669970 15230 leader_election.cc:302] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 81409d6217b64c2794dca75da4b7e534; no voters: c81d9bc77ee54b4c8d04746984c5ed6e
I20260928 13:08:33.670039 15985 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:08:34.504209 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:34.503786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:36257)
0928 13:08:34.504112 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:34.504156 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
I20260928 13:08:34.504269 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:34.503638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:48061)
0928 13:08:34.503991 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:34.504039 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":65,"threads_started":1}
I20260928 13:08:34.504387 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:34.503759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:46853)
0928 13:08:34.504317 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:34.504357 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":60,"threads_started":1}
I20260928 13:08:35.505194 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:35.504891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:57245)
0928 13:08:35.505091 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:35.505137 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":58,"threads_started":1}
I20260928 13:08:35.505357 15992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:35.504918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:57853)
0928 13:08:35.505264 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:35.505303 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":47,"threads_started":1}
I20260928 13:08:35.505419 15991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:35.504890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:35033)
0928 13:08:35.505327 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:35.505361 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":65,"threads_started":1}
I20260928 13:08:36.506201 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:36.505916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:39189)
0928 13:08:36.506087 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:36.506132 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20260928 13:08:36.506299 15996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:36.506110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:51185)
0928 13:08:36.506207 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:36.506246 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":47,"threads_started":1}
I20260928 13:08:36.506321 15994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:36.505904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:58459)
0928 13:08:36.506235 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:36.506266 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":68,"threads_started":1}
I20260928 13:08:37.507036 15998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:37.506832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:44287)
0928 13:08:37.506936 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:37.506977 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":43,"threads_started":1}
I20260928 13:08:37.507043 15997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:37.506769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:46051)
0928 13:08:37.506949 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:37.506991 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":92,"threads_started":1}
I20260928 13:08:37.507079 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:37.506812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:50665)
0928 13:08:37.507004 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:37.507034 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
I20260928 13:08:38.508155 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:38.507620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:57971)
0928 13:08:38.508007 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:38.508072 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":90,"threads_started":1}
I20260928 13:08:38.508520 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:38.507698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:52525)
0928 13:08:38.508218 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:38.508265 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":55,"threads_started":1}
I20260928 13:08:38.508658 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:38.508142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:57171)
0928 13:08:38.508560 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:38.508601 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":99,"threads_started":1}
I20260928 13:08:38.781028 16003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c81d9bc77ee54b4c8d04746984c5ed6e)
I20260928 13:08:38.781136 16003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } }
I20260928 13:08:38.781292 16003 leader_election.cc:288] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479)
I20260928 13:08:38.781770 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:38.781434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:54452)
0928 13:08:38.781683 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:38.781726 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":95,"threads_started":1}
W20260928 13:08:38.781895 15230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260928 13:08:38.782334 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:38.782219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:54460)
0928 13:08:38.782280 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:38.782311 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 13:08:38.782437 15230 leader_election.cc:334] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479): Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
I20260928 13:08:38.782475 15230 leader_election.cc:302] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 81409d6217b64c2794dca75da4b7e534; no voters: c81d9bc77ee54b4c8d04746984c5ed6e
I20260928 13:08:38.782582 16003 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:08:39.509248 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:39.508856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:39935)
0928 13:08:39.509133 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:39.509180 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":92,"threads_started":1}
I20260928 13:08:39.509545 16008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:39.509246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:39267)
0928 13:08:39.509398 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:39.509490 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260928 13:08:39.509577 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:39.509099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:51083)
0928 13:08:39.509491 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:39.509527 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":56,"threads_started":1}
I20260928 13:08:40.510455 16009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:40.509967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:35057)
0928 13:08:40.510360 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:40.510401 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":79,"threads_started":1}
I20260928 13:08:40.510592 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:40.510186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:57273)
0928 13:08:40.510351 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:40.510401 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":54,"threads_started":1}
I20260928 13:08:40.510643 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:40.510309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:33251)
0928 13:08:40.510482 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:40.510519 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":44,"threads_started":1}
I20260928 13:08:41.511431 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:41.511208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:47125)
0928 13:08:41.511333 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:41.511373 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260928 13:08:41.511583 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:41.511174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:60195)
0928 13:08:41.511482 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:41.511520 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":68,"threads_started":1}
I20260928 13:08:41.511688 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:41.511373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:33325)
0928 13:08:41.511523 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:41.511643 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":46,"threads_started":1}
I20260928 13:08:42.512255 16016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:42.512037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:42939)
0928 13:08:42.512152 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:42.512194 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":45,"threads_started":1}
I20260928 13:08:42.512333 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:42.512144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:54321)
0928 13:08:42.512255 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:42.512290 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260928 13:08:42.512410 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:42.511933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:44701)
0928 13:08:42.512270 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:42.512357 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":84,"threads_started":1}
I20260928 13:08:43.513398 16018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:43.513161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:43381)
0928 13:08:43.513319 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:43.513351 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":101,"threads_started":1}
I20260928 13:08:43.513398 16019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:43.513161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:54253)
0928 13:08:43.513295 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:43.513345 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260928 13:08:43.513571 16020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:43.513380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:56495)
0928 13:08:43.513486 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:43.513520 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":48,"threads_started":1}
I20260928 13:08:44.514604 16022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:44.514313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:48613)
0928 13:08:44.514464 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:44.514522 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260928 13:08:44.514667 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:44.514222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:36499)
0928 13:08:44.514580 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:44.514613 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":90,"threads_started":1}
I20260928 13:08:44.514863 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:44.514523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:56199)
0928 13:08:44.514670 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:44.514754 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260928 13:08:45.515561 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:45.515304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:43105)
0928 13:08:45.515459 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:45.515500 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260928 13:08:45.515561 16024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:45.515304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:57247)
0928 13:08:45.515468 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:45.515511 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20260928 13:08:45.515808 16026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:45.515353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:42755)
0928 13:08:45.515678 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:45.515757 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":89,"threads_started":1}
I20260928 13:08:46.516367 16027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:46.516079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:35525)
0928 13:08:46.516246 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:46.516292 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":78,"threads_started":1}
I20260928 13:08:46.516566 16028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:46.516126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:49927)
0928 13:08:46.516469 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:46.516514 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":42,"threads_started":1}
I20260928 13:08:46.516587 16029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:46.516376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:36839)
0928 13:08:46.516508 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:46.516535 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260928 13:08:47.517350 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:47.517003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:36371)
0928 13:08:47.517238 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:47.517292 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":71,"threads_started":1}
I20260928 13:08:47.517482 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:47.517301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:50105)
0928 13:08:47.517404 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:47.517433 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":41,"threads_started":1}
I20260928 13:08:47.517488 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:47.517240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:52179)
0928 13:08:47.517382 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:47.517433 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260928 13:08:48.518167 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:48.517862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:56205)
0928 13:08:48.518031 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:48.518107 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":94,"threads_started":1}
I20260928 13:08:48.518268 16034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:48.517909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:43837)
0928 13:08:48.518191 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:48.518228 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":51,"threads_started":1}
I20260928 13:08:48.518306 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:48.518099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:47455)
0928 13:08:48.518200 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:48.518235 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":42,"threads_started":1}
I20260928 13:08:49.518965 16037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:49.518735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:35033)
0928 13:08:49.518859 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:49.518903 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":42,"threads_started":1}
I20260928 13:08:49.518992 16038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:49.518805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:57845)
0928 13:08:49.518906 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:49.518945 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":57,"threads_started":1}
I20260928 13:08:49.519192 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:49.518631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:52503)
0928 13:08:49.519097 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:49.519132 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":70,"threads_started":1}
I20260928 13:08:50.519860 16039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:50.519470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:43591)
0928 13:08:50.519748 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:50.519799 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":67,"threads_started":1}
I20260928 13:08:50.520334 16040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:50.519644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:43215)
0928 13:08:50.520220 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:50.520286 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":54,"threads_started":1}
I20260928 13:08:50.520365 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:50.519644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:33059)
0928 13:08:50.520270 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:50.520313 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":58,"threads_started":1}
I20260928 13:08:51.521142 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:51.520503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:40051)
0928 13:08:51.521010 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:51.521081 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":75,"threads_started":1}
I20260928 13:08:51.521329 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:51.521085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:58265)
0928 13:08:51.521242 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:51.521285 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260928 13:08:51.521337 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:51.521053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:52269)
0928 13:08:51.521234 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:51.521284 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":64,"threads_started":1}
I20260928 13:08:52.522161 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:52.521900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:60239)
0928 13:08:52.522053 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:52.522099 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260928 13:08:52.522213 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:52.522022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:53827)
0928 13:08:52.522128 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:52.522168 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":49,"threads_started":1}
I20260928 13:08:52.522518 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:52.521889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:58199)
0928 13:08:52.522428 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:52.522469 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":66,"threads_started":1}
I20260928 13:08:53.523108 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:53.522891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.130:32903)
0928 13:08:53.523016 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:53.523054 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260928 13:08:53.523155 16049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:53.522880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.131:50133)
0928 13:08:53.523015 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:53.523054 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260928 13:08:53.523224 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:53.522896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.8.88.129:46435)
0928 13:08:53.523116 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:53.523156 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":54,"threads_started":1}
I20260928 13:08:53.568996 16051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c81d9bc77ee54b4c8d04746984c5ed6e)
I20260928 13:08:53.569118 16051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c81d9bc77ee54b4c8d04746984c5ed6e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43479 } } peers { permanent_uuid: "81409d6217b64c2794dca75da4b7e534" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37155 } attrs { promote: false } } peers { permanent_uuid: "5626e470e9c945789fb7f753593335a0" member_type: NON_VOTER last_known_addr { host: "127.8.88.188" port: 45571 } attrs { promote: true } }
I20260928 13:08:53.569244 16051 leader_election.cc:288] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479)
I20260928 13:08:53.569624 16052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:53.569411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:57498)
0928 13:08:53.569538 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:53.569577 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
W20260928 13:08:53.569716 15230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
I20260928 13:08:53.570236 16052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:53.570130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:43479 (local address 127.0.0.1:57502)
0928 13:08:53.570186 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:53.570216 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 13:08:53.570358 15230 leader_election.cc:334] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c81d9bc77ee54b4c8d04746984c5ed6e (127.8.88.190:43479): Network error: Client connection negotiation failed: client connection to 127.8.88.190:43479: connect: Connection refused (error 111)
I20260928 13:08:53.570434 15230 leader_election.cc:302] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 81409d6217b64c2794dca75da4b7e534; no voters: c81d9bc77ee54b4c8d04746984c5ed6e
I20260928 13:08:53.570631 16051 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 81409d6217b64c2794dca75da4b7e534 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.8.88.190:43479,127.8.88.189:37155,127.8.88.188:45571, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.8.88.190:43479,127.8.88.189:37155,127.8.88.188:45571, num_attempts: 1)
I20260928 13:08:53.734711 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 15390
I20260928 13:08:53.740130 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 15617
I20260928 13:08:53.745317 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 15753
I20260928 13:08:53.750833 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 15110
I20260928 13:08:53.755880 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 15297
2026-09-28T13:08:53Z chronyd exiting
I20260928 13:08:53.772403 8546 test_util.cc:175] -----------------------------------------------
I20260928 13:08:53.772500 8546 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790600874459249-8546-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33342 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:08:53 dist-test-slave-p1lc krb5kdc[16056](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:08:53 dist-test-slave-p1lc krb5kdc[16056](info): set up 2 sockets
Sep 28 13:08:53 dist-test-slave-p1lc krb5kdc[16056](info): commencing operation
krb5kdc: starting...
W20260928 13:08:54.825888 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:08:54 dist-test-slave-p1lc krb5kdc[16056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:08:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.190.keytab.
Entry for principal oryx/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.190.keytab.
I20260928 13:08:54.964805 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46851
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.190.keytab
--principal=oryx/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:55.053562 16069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:55.053797 16069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:55.053820 16069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:55.055085 16069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:55.055141 16069 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 13:08:55.055155 16069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:55.055168 16069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:55.055181 16069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:55.055194 16069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:55.056926 16069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46851
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.190.keytab
--principal=oryx/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:55.057232 16069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:55.057479 16069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:55.063423 16075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:55.063616 16069 server_base.cc:1034] running on GCE node
W20260928 13:08:55.063418 16074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:55.063418 16077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:55.064147 16069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:55.064525 16069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:55.065692 16069 hybrid_clock.cc:648] HybridClock initialized: now 1790600935065670 us; error 47 us; skew 500 ppm
Sep 28 13:08:55 dist-test-slave-p1lc krb5kdc[16056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600935, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:55.068480 16069 init.cc:377] Logged in from keytab as oryx/127.8.88.190@KRBTEST.COM (short username oryx)
I20260928 13:08:55.069078 16069 webserver.cc:460] Webserver started at http://127.8.88.190:44933/ using document root <none> and password file <none>
I20260928 13:08:55.069285 16069 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:55.069327 16069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:55.069419 16069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:55.070501 16069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "6e6d32b09b1d49188fbb4d026acaf982"
format_stamp: "Formatted at 2026-09-28 13:08:55 on dist-test-slave-p1lc"
server_key: "d32a5387acd6a4817e0a01105a3524d4"
server_key_iv: "d1ebc804a4c52d93c316ee419e1f47d8"
server_key_version: "encryptionkey@0"
I20260928 13:08:55.070819 16069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "6e6d32b09b1d49188fbb4d026acaf982"
format_stamp: "Formatted at 2026-09-28 13:08:55 on dist-test-slave-p1lc"
server_key: "d32a5387acd6a4817e0a01105a3524d4"
server_key_iv: "d1ebc804a4c52d93c316ee419e1f47d8"
server_key_version: "encryptionkey@0"
I20260928 13:08:55.072228 16069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:08:55.073148 16084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:55.073402 16069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:55.073488 16069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "6e6d32b09b1d49188fbb4d026acaf982"
format_stamp: "Formatted at 2026-09-28 13:08:55 on dist-test-slave-p1lc"
server_key: "d32a5387acd6a4817e0a01105a3524d4"
server_key_iv: "d1ebc804a4c52d93c316ee419e1f47d8"
server_key_version: "encryptionkey@0"
I20260928 13:08:55.073694 16069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:55.083787 16069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:55.086359 16069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:55.086522 16069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:55.091786 16069 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:46851
I20260928 13:08:55.091861 16146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:46851 every 8 connection(s)
I20260928 13:08:55.092242 16069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:55.093165 16147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:55.095603 16147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982: Bootstrap starting.
I20260928 13:08:55.096253 16147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:55.096570 16147 log.cc:824] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:55.097530 16147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982: No bootstrap required, opened a new log
I20260928 13:08:55.100185 16147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6d32b09b1d49188fbb4d026acaf982" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46851 } }
I20260928 13:08:55.100359 16147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:55.100387 16147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6d32b09b1d49188fbb4d026acaf982, State: Initialized, Role: FOLLOWER
I20260928 13:08:55.100571 16147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6d32b09b1d49188fbb4d026acaf982" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46851 } }
I20260928 13:08:55.100675 16147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:55.100739 16147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:55.100793 16147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:55.100857 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 16069
I20260928 13:08:55.100991 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:55.101186 8546 external_mini_cluster.cc:1428] Setting key f90079ad86fc8eab54202b3a701f0efe
I20260928 13:08:55.101739 16147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6d32b09b1d49188fbb4d026acaf982" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46851 } }
I20260928 13:08:55.101930 16147 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e6d32b09b1d49188fbb4d026acaf982; no voters:
I20260928 13:08:55.102144 16147 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:55.102187 16152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:55.102413 16152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [term 1 LEADER]: Becoming Leader. State: Replica: 6e6d32b09b1d49188fbb4d026acaf982, State: Running, Role: LEADER
I20260928 13:08:55.102473 16147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:55.102592 16152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6d32b09b1d49188fbb4d026acaf982" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46851 } }
I20260928 13:08:55.103073 16153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6d32b09b1d49188fbb4d026acaf982. Latest consensus state: current_term: 1 leader_uuid: "6e6d32b09b1d49188fbb4d026acaf982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6d32b09b1d49188fbb4d026acaf982" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46851 } } }
I20260928 13:08:55.103276 16153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:55.103708 16161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:55.104302 16154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e6d32b09b1d49188fbb4d026acaf982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6d32b09b1d49188fbb4d026acaf982" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46851 } } }
I20260928 13:08:55.104485 16154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:55.104708 16161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:55.106724 16161 catalog_manager.cc:1347] Generated new cluster ID: 5d2f37096a2c40768ac84a4bc24e38f8
I20260928 13:08:55.106782 16161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:08:55 dist-test-slave-p1lc krb5kdc[16056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
I20260928 13:08:55.114607 16161 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:55.115139 16161 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:55.116925 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.101979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52862 (local address 127.8.88.190:46851)
0928 13:08:55.102198 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.102210 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.102257 (+ 47us) server_negotiation.cc:407] Connection header received
0928 13:08:55.102410 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.102417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.102495 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.102656 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:55.103504 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.105536 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.108969 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.109661 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.109793 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.112904 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:55.112910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:55.112910 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:55.112926 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:55.115145 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.115588 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.115592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.115592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.115712 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.115951 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.115952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.115952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.116232 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:55.116382 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.116513 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.116657 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":119,"threads_started":1}
I20260928 13:08:55.120554 16161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982: Generated new TSK 0
I20260928 13:08:55.120774 16161 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab.
Entry for principal oryx/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab.
I20260928 13:08:55.153970 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46851
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab
--principal=oryx/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:55.238952 16173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:55.239225 16173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:55.239271 16173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:55.240878 16173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:55.240995 16173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:55.241034 16173 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 13:08:55.241103 16173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:08:55.243614 16173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab
--principal=oryx/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.88.190:46851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:55.243991 16173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:55.244244 16173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:55.249922 16178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:55.249959 16173 server_base.cc:1034] running on GCE node
W20260928 13:08:55.249940 16179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:55.249956 16181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:55.250504 16173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:55.250779 16173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:55.251977 16173 hybrid_clock.cc:648] HybridClock initialized: now 1790600935251977 us; error 45 us; skew 500 ppm
Sep 28 13:08:55 dist-test-slave-p1lc krb5kdc[16056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600935, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:55.254956 16173 init.cc:377] Logged in from keytab as oryx/127.8.88.129@KRBTEST.COM (short username oryx)
I20260928 13:08:55.255465 16173 webserver.cc:460] Webserver started at http://127.8.88.129:35027/ using document root <none> and password file <none>
I20260928 13:08:55.255724 16173 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:55.255800 16173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:55.255954 16173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:55.257153 16173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "9127714d0c1940198ed545e9565ef40f"
format_stamp: "Formatted at 2026-09-28 13:08:55 on dist-test-slave-p1lc"
server_key: "b4b7fab32b9bd9f6b145a1a8ffaffcab"
server_key_iv: "bf3a86bfb19d3423cf0c00d8bed24e60"
server_key_version: "encryptionkey@0"
I20260928 13:08:55.257548 16173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "9127714d0c1940198ed545e9565ef40f"
format_stamp: "Formatted at 2026-09-28 13:08:55 on dist-test-slave-p1lc"
server_key: "b4b7fab32b9bd9f6b145a1a8ffaffcab"
server_key_iv: "bf3a86bfb19d3423cf0c00d8bed24e60"
server_key_version: "encryptionkey@0"
I20260928 13:08:55.258983 16173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:08:55.259923 16188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:55.260170 16173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:55.260293 16173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "9127714d0c1940198ed545e9565ef40f"
format_stamp: "Formatted at 2026-09-28 13:08:55 on dist-test-slave-p1lc"
server_key: "b4b7fab32b9bd9f6b145a1a8ffaffcab"
server_key_iv: "bf3a86bfb19d3423cf0c00d8bed24e60"
server_key_version: "encryptionkey@0"
I20260928 13:08:55.260389 16173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:55.269883 16173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:55.272192 16173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:55.272385 16173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:55.272986 16173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:55.273054 16173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:55.273098 16173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:55.273126 16173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 13:08:55 dist-test-slave-p1lc krb5kdc[16056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600935, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
I20260928 13:08:55.280692 16173 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:39089
I20260928 13:08:55.280826 16303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:39089 every 8 connection(s)
I20260928 13:08:55.281144 16173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:08:55.284557 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.273510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:37731 (local address 127.8.88.190:46851)
0928 13:08:55.273591 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.273595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.273764 (+ 169us) server_negotiation.cc:407] Connection header received
0928 13:08:55.274273 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.274275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.274289 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.274372 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:55.275354 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.276204 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.277642 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.277993 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.278072 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.281264 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:55.281266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:55.281267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:55.281280 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:55.283083 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.283521 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.283523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.283524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.283598 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.283873 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.283874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.283875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.284080 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:55.284146 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.284416 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.284489 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 13:08:55.284754 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.273478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46851 (local address 127.8.88.129:37731)
0928 13:08:55.273632 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.273673 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.274157 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.274528 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.274531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.274912 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:55.275255 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.275270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.276294 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.276295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.277526 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.277529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.278111 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.278112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.278314 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.278887 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:55.278900 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:55.281120 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:55.283180 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.283186 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.283188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.283428 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.283671 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.283672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.283672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.283778 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.284151 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:55.284157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:55.284289 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.284380 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.284523 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260928 13:08:55.285393 16304 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46851
I20260928 13:08:55.285475 16304 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:55.285705 16304 heartbeater.cc:511] Master 127.8.88.190:46851 requested a full tablet report, sending...
I20260928 13:08:55.286288 16101 ts_manager.cc:194] Registered new tserver with Master: 9127714d0c1940198ed545e9565ef40f (127.8.88.129:39089)
I20260928 13:08:55.287072 16101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:37731
I20260928 13:08:55.290017 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 16173
I20260928 13:08:55.290133 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:08:55.290356 8546 external_mini_cluster.cc:1428] Setting key 9e9dd09901b1f3dc9b6f8b82d585d681
I20260928 13:08:55.293666 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.286710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:40375 (local address 127.8.88.190:46851)
0928 13:08:55.286753 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.286755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.286954 (+ 199us) server_negotiation.cc:407] Connection header received
0928 13:08:55.286978 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.286979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.286993 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.287072 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:55.287518 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.288012 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.288723 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.289082 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.289214 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.290426 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:55.290428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:55.290428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:55.290436 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:55.292246 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.292740 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.292741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.292741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.292794 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.293107 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.293107 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.293108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.293220 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:55.293279 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.293492 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.293597 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:08:55.293781 16310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.286570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46851 (local address 127.8.88.129:40375)
0928 13:08:55.286797 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.286809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.286881 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.287080 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.287081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.287306 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:55.287429 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.287432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.288109 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.288110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.288633 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.288635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.289218 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.289219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.289298 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.289845 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:55.289850 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:55.290296 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:55.292351 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.292352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.292353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.292656 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.292942 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.292943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.292944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.293031 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.293305 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:55.293306 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:55.293376 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.293466 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.293626 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for oryx/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab.
Entry for principal oryx/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab.
I20260928 13:08:55.321018 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46851
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab
--principal=oryx/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:55.406519 16313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:55.406795 16313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:55.406837 16313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:55.408094 16313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:55.408174 16313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:55.408211 16313 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 13:08:55.408253 16313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:08:55.409920 16313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab
--principal=oryx/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.88.190:46851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:55.410233 16313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:55.410481 16313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:08:55.416239 16313 server_base.cc:1034] running on GCE node
W20260928 13:08:55.416254 16318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:55.416234 16321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:55.416448 16319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:55.416677 16313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:55.416987 16313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:55.418146 16313 hybrid_clock.cc:648] HybridClock initialized: now 1790600935418125 us; error 36 us; skew 500 ppm
Sep 28 13:08:55 dist-test-slave-p1lc krb5kdc[16056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600935, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:55.421020 16313 init.cc:377] Logged in from keytab as oryx/127.8.88.130@KRBTEST.COM (short username oryx)
I20260928 13:08:55.421492 16313 webserver.cc:460] Webserver started at http://127.8.88.130:45893/ using document root <none> and password file <none>
I20260928 13:08:55.421739 16313 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:55.421815 16313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:55.421975 16313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:55.423218 16313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "3153063587854129a678e89dd6c06754"
format_stamp: "Formatted at 2026-09-28 13:08:55 on dist-test-slave-p1lc"
server_key: "74e9b96a17081886f56121eb3ce21258"
server_key_iv: "7cc654ffefafca4431fa381e416bf7d1"
server_key_version: "encryptionkey@0"
I20260928 13:08:55.423627 16313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "3153063587854129a678e89dd6c06754"
format_stamp: "Formatted at 2026-09-28 13:08:55 on dist-test-slave-p1lc"
server_key: "74e9b96a17081886f56121eb3ce21258"
server_key_iv: "7cc654ffefafca4431fa381e416bf7d1"
server_key_version: "encryptionkey@0"
I20260928 13:08:55.425187 16313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:08:55.426242 16328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:55.426517 16313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:55.426637 16313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "3153063587854129a678e89dd6c06754"
format_stamp: "Formatted at 2026-09-28 13:08:55 on dist-test-slave-p1lc"
server_key: "74e9b96a17081886f56121eb3ce21258"
server_key_iv: "7cc654ffefafca4431fa381e416bf7d1"
server_key_version: "encryptionkey@0"
I20260928 13:08:55.426718 16313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:55.456389 16313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:55.459309 16313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:55.459537 16313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:55.460198 16313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:55.460271 16313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:55.460319 16313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:55.460345 16313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 13:08:55 dist-test-slave-p1lc krb5kdc[16056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600935, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
I20260928 13:08:55.468221 16313 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:42967
I20260928 13:08:55.468277 16443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:42967 every 8 connection(s)
I20260928 13:08:55.468614 16313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:08:55.471764 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.460684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:43059 (local address 127.8.88.190:46851)
0928 13:08:55.460760 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.460762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.461048 (+ 286us) server_negotiation.cc:407] Connection header received
0928 13:08:55.461654 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.461655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.461691 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.461781 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:55.462754 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.463640 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.465018 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.465231 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.465323 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.467983 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:55.467985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:55.467985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:55.467995 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:55.470132 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.470599 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.470600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.470601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.470662 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.471031 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.471032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.471033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.471163 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:55.471235 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.471602 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.471695 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 13:08:55.471905 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.460692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46851 (local address 127.8.88.130:43059)
0928 13:08:55.460904 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.460950 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.461486 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.461885 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.461890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.462227 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:55.462628 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.462637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.463767 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.463768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.464866 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.464870 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.465358 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.465359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.465498 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.466091 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:55.466101 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:55.467842 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:55.470245 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.470252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.470254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.470512 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.470756 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.470757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.470758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.470938 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.471268 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:55.471274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:55.471426 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.471538 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.471688 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
I20260928 13:08:55.472489 16444 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46851
I20260928 13:08:55.472585 16444 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:55.472792 16444 heartbeater.cc:511] Master 127.8.88.190:46851 requested a full tablet report, sending...
I20260928 13:08:55.473394 16100 ts_manager.cc:194] Registered new tserver with Master: 3153063587854129a678e89dd6c06754 (127.8.88.130:42967)
I20260928 13:08:55.474288 16100 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:43059
I20260928 13:08:55.476917 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 16313
I20260928 13:08:55.477039 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:08:55.477273 8546 external_mini_cluster.cc:1428] Setting key 5ec393403d2232acdf4b0bc116c83872
I20260928 13:08:55.480717 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.473718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46851 (local address 127.8.88.130:34681)
0928 13:08:55.473866 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.473880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.473952 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.474148 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.474149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.474377 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:55.474461 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.474464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.475293 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.475294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.475831 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.475833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.476362 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.476363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.476437 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.477177 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:55.477184 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:55.477641 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:55.479424 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.479425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.479426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.479685 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.479938 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.479939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.479939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.480008 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.480416 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:55.480417 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:55.480467 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.480540 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.480639 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260928 13:08:55.480784 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.473768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:34681 (local address 127.8.88.190:46851)
0928 13:08:55.473815 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.473817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.473952 (+ 135us) server_negotiation.cc:407] Connection header received
0928 13:08:55.474055 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.474056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.474065 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.474115 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:55.474552 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.475116 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.475942 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.476235 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.476363 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.477732 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:55.477734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:55.477734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:55.477741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:55.479331 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.479787 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.479789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.479789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.479864 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.480073 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.480074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.480075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.480316 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:55.480396 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.480585 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.480684 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for oryx/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab.
Entry for principal oryx/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab.
I20260928 13:08:55.509330 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46851
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab
--principal=oryx/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:55.597438 16453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:55.597656 16453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:55.597683 16453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:55.598955 16453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:55.599007 16453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:55.599030 16453 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 13:08:55.599047 16453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:08:55.600632 16453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab
--principal=oryx/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.88.190:46851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:55.600941 16453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:55.601207 16453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:55.607128 16458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:55.607180 16453 server_base.cc:1034] running on GCE node
W20260928 13:08:55.607138 16461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:55.607144 16459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:55.607628 16453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:55.607930 16453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:55.609149 16453 hybrid_clock.cc:648] HybridClock initialized: now 1790600935609114 us; error 41 us; skew 500 ppm
Sep 28 13:08:55 dist-test-slave-p1lc krb5kdc[16056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600935, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:55.612187 16453 init.cc:377] Logged in from keytab as oryx/127.8.88.131@KRBTEST.COM (short username oryx)
I20260928 13:08:55.612788 16453 webserver.cc:460] Webserver started at http://127.8.88.131:45387/ using document root <none> and password file <none>
I20260928 13:08:55.613096 16453 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:55.613150 16453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:55.613297 16453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:55.614663 16453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "811f434ce8b9435c924097885cbe163e"
format_stamp: "Formatted at 2026-09-28 13:08:55 on dist-test-slave-p1lc"
server_key: "c401a60ae00fc5ec837e71785c906076"
server_key_iv: "df1d6c5c775d1e527d7d22934fb6cf19"
server_key_version: "encryptionkey@0"
I20260928 13:08:55.615090 16453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "811f434ce8b9435c924097885cbe163e"
format_stamp: "Formatted at 2026-09-28 13:08:55 on dist-test-slave-p1lc"
server_key: "c401a60ae00fc5ec837e71785c906076"
server_key_iv: "df1d6c5c775d1e527d7d22934fb6cf19"
server_key_version: "encryptionkey@0"
I20260928 13:08:55.616623 16453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:08:55.617676 16468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:55.618170 16453 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:55.618265 16453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "811f434ce8b9435c924097885cbe163e"
format_stamp: "Formatted at 2026-09-28 13:08:55 on dist-test-slave-p1lc"
server_key: "c401a60ae00fc5ec837e71785c906076"
server_key_iv: "df1d6c5c775d1e527d7d22934fb6cf19"
server_key_version: "encryptionkey@0"
I20260928 13:08:55.618350 16453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:55.635056 16453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:55.637418 16453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:55.637580 16453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:55.638214 16453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:55.638281 16453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:55.638324 16453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:55.638342 16453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:55.646245 16453 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:39755
Sep 28 13:08:55 dist-test-slave-p1lc krb5kdc[16056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600935, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
I20260928 13:08:55.646495 16583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:39755 every 8 connection(s)
I20260928 13:08:55.646757 16453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:08:55.650198 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.638687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:58535 (local address 127.8.88.190:46851)
0928 13:08:55.638763 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.638765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.639054 (+ 289us) server_negotiation.cc:407] Connection header received
0928 13:08:55.639896 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.639898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.639912 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.639985 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:55.641057 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.641584 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.643487 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.643711 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.643827 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.647203 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:55.647206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:55.647207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:55.647217 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:55.648785 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.649336 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.649338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.649338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.649396 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.649639 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.649639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.649640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.649754 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:55.649802 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.650065 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.650132 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 13:08:55.650369 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.638678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46851 (local address 127.8.88.131:58535)
0928 13:08:55.638881 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.638943 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.639716 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.640132 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.640136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.640580 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:55.640944 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.640952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.641674 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.641676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.643367 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.643371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.643853 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.643855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.644008 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.644953 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:55.644970 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:55.647057 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:55.648986 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.648992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.648994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.649241 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.649464 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.649464 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.649465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.649572 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.649807 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:55.649811 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:55.649941 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.650022 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.650156 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260928 13:08:55.650923 16584 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46851
I20260928 13:08:55.651010 16584 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:55.651203 16584 heartbeater.cc:511] Master 127.8.88.190:46851 requested a full tablet report, sending...
I20260928 13:08:55.651686 16101 ts_manager.cc:194] Registered new tserver with Master: 811f434ce8b9435c924097885cbe163e (127.8.88.131:39755)
I20260928 13:08:55.652196 16101 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:58535
I20260928 13:08:55.655074 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 16453
I20260928 13:08:55.655195 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:08:55.655385 8546 external_mini_cluster.cc:1428] Setting key ee2b8c20ca25efc6a9545b5276ba4a5c
I20260928 13:08:55.656227 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:08:55.657999 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.652119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:33807 (local address 127.8.88.190:46851)
0928 13:08:55.652162 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.652164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.652224 (+ 60us) server_negotiation.cc:407] Connection header received
0928 13:08:55.652293 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.652294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.652305 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.652362 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:55.652837 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.653343 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.654081 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.654273 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.654350 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.655315 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:55.655317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:55.655317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:55.655324 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:55.656801 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.657207 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.657209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.657209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.657269 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.657475 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.657477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.657477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.657594 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:55.657674 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.657826 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.657906 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:08:55.658025 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.652026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46851 (local address 127.8.88.131:33807)
0928 13:08:55.652165 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.652176 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.652240 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.652515 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.652516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.652685 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:55.652745 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.652747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.653446 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.653447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.653978 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.653980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.654348 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.654349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.654414 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.654894 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:55.654899 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:55.655241 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:55.656919 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.656920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.656921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.657113 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.657337 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.657338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.657338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.657398 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.657664 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:55.657665 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:55.657715 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.657802 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.657908 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
Sep 28 13:08:55 dist-test-slave-p1lc krb5kdc[16056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600935, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 13:08:55 dist-test-slave-p1lc krb5kdc[16056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:08:55 dist-test-slave-p1lc krb5kdc[16056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 13:08:55.679888 16151 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:08:55.673900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52868 (local address 127.8.88.190:46851)
0928 13:08:55.673980 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.673983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.674207 (+ 224us) server_negotiation.cc:407] Connection header received
0928 13:08:55.674316 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.674317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.674335 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.674415 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:55.675028 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.675543 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.676315 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.676525 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.676612 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.679798 (+ 3186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52868: BlockingRecv error: recv got EOF from 127.0.0.1:52868 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 13:08:55 dist-test-slave-p1lc krb5kdc[16056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600935, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
I20260928 13:08:55.688316 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.681002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.8.88.190:46851)
0928 13:08:55.681087 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.681089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.681120 (+ 31us) server_negotiation.cc:407] Connection header received
0928 13:08:55.681187 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.681188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.681202 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.681254 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:55.681814 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.682385 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.683127 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.683318 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.683475 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.685802 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:55.685803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:55.685804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:55.685811 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:55.687343 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.687699 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.687700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.687700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.687753 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.687916 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.687917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.687917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.688024 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:55.688098 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.688188 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.688247 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 13:08:55.690306 16101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:08:55.692000 16101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:55.699465 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.694680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:42967 (local address 127.0.0.1:40224)
0928 13:08:55.694873 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.694920 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.695064 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.695416 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.695420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.695422 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:55.695507 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.695510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.696468 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.696470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.697988 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.697991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.699077 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.699078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.699202 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.699220 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.699314 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.699370 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260928 13:08:55.699496 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.694781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.8.88.130:42967)
0928 13:08:55.695184 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.695191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.695202 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:08:55.695267 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.695269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.695285 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.695593 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:55.695661 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.696355 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.698066 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.698912 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.698995 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.699071 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.699346 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.699391 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":76,"threads_started":1}
I20260928 13:08:55.700354 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.694900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:39089 (local address 127.0.0.1:49484)
0928 13:08:55.695050 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.695060 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.695105 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.695504 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.695504 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.695506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:55.695554 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.695556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.696850 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.696851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.697996 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.697997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.700081 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.700082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.700139 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.700143 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.700216 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.700264 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":32,"thread_start_us":90,"threads_started":1}
I20260928 13:08:55.700584 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.694691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:39755 (local address 127.0.0.1:50458)
0928 13:08:55.694936 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.694949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.695010 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.696747 (+ 1737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.696749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.696751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:55.696791 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.696792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.697970 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.697971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.699086 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.699088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.700354 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.700356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.700434 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.700438 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.700469 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.700510 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":83,"threads_started":1}
I20260928 13:08:55.700661 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.695127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.8.88.129:39089)
0928 13:08:55.695328 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.695334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.695344 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:08:55.695383 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.695385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.695401 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.695486 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:55.695631 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.696354 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.699118 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.699906 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.699969 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.700047 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.700257 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.700297 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260928 13:08:55.700778 16378 tablet_service.cc:1467] Processing CreateTablet for tablet f71d05209d554e6c8d0d71de3b493f5c (DEFAULT_TABLE table=test-table [id=e31ecdb76fb64ca6a4b3a50afbadf9eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:55.701043 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.694780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.8.88.131:39755)
0928 13:08:55.696558 (+ 1778us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.696563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.696574 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:08:55.696611 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.696614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.696631 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.696747 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:55.696995 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.697885 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.699207 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.700012 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.700762 (+ 750us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.700849 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.700906 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.700950 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1738,"thread_start_us":83,"threads_started":1}
I20260928 13:08:55.701253 16378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71d05209d554e6c8d0d71de3b493f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:55.701393 16234 tablet_service.cc:1467] Processing CreateTablet for tablet f71d05209d554e6c8d0d71de3b493f5c (DEFAULT_TABLE table=test-table [id=e31ecdb76fb64ca6a4b3a50afbadf9eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:55.701615 16518 tablet_service.cc:1467] Processing CreateTablet for tablet f71d05209d554e6c8d0d71de3b493f5c (DEFAULT_TABLE table=test-table [id=e31ecdb76fb64ca6a4b3a50afbadf9eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:55.701740 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71d05209d554e6c8d0d71de3b493f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:55.701956 16518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71d05209d554e6c8d0d71de3b493f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:55.705308 16612 tablet_bootstrap.cc:492] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754: Bootstrap starting.
I20260928 13:08:55.706104 16613 tablet_bootstrap.cc:492] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e: Bootstrap starting.
I20260928 13:08:55.706005 16612 tablet_bootstrap.cc:654] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:55.706627 16612 log.cc:824] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:55.706782 16613 tablet_bootstrap.cc:654] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:55.707140 16613 log.cc:824] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:55.707528 16612 tablet_bootstrap.cc:492] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754: No bootstrap required, opened a new log
I20260928 13:08:55.707548 16614 tablet_bootstrap.cc:492] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f: Bootstrap starting.
I20260928 13:08:55.707635 16612 ts_tablet_manager.cc:1397] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 13:08:55.707950 16613 tablet_bootstrap.cc:492] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e: No bootstrap required, opened a new log
I20260928 13:08:55.708022 16613 ts_tablet_manager.cc:1397] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:55.708177 16614 tablet_bootstrap.cc:654] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:55.708506 16614 log.cc:824] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:55.709352 16614 tablet_bootstrap.cc:492] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f: No bootstrap required, opened a new log
I20260928 13:08:55.709439 16614 ts_tablet_manager.cc:1397] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:55.709817 16612 raft_consensus.cc:348] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.710011 16612 raft_consensus.cc:374] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:55.710057 16612 raft_consensus.cc:729] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3153063587854129a678e89dd6c06754, State: Initialized, Role: FOLLOWER
I20260928 13:08:55.710184 16612 consensus_queue.cc:260] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.710498 16612 ts_tablet_manager.cc:1428] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:08:55.710526 16444 heartbeater.cc:503] Master 127.8.88.190:46851 was elected leader, sending a full tablet report...
I20260928 13:08:55.710356 16613 raft_consensus.cc:348] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.710846 16613 raft_consensus.cc:374] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:55.710889 16613 raft_consensus.cc:729] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811f434ce8b9435c924097885cbe163e, State: Initialized, Role: FOLLOWER
I20260928 13:08:55.711023 16613 consensus_queue.cc:260] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.711313 16613 ts_tablet_manager.cc:1428] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:08:55.711345 16584 heartbeater.cc:503] Master 127.8.88.190:46851 was elected leader, sending a full tablet report...
I20260928 13:08:55.711588 16614 raft_consensus.cc:348] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.711748 16614 raft_consensus.cc:374] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:55.711789 16614 raft_consensus.cc:729] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9127714d0c1940198ed545e9565ef40f, State: Initialized, Role: FOLLOWER
I20260928 13:08:55.711905 16614 consensus_queue.cc:260] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.712179 16614 ts_tablet_manager.cc:1428] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:08:55.712255 16304 heartbeater.cc:503] Master 127.8.88.190:46851 was elected leader, sending a full tablet report...
W20260928 13:08:55.719280 16445 tablet.cc:2369] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:08:55.781993 16305 tablet.cc:2369] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:08:55.860008 16619 raft_consensus.cc:482] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:55.860183 16619 raft_consensus.cc:504] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.860594 16619 leader_election.cc:288] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3153063587854129a678e89dd6c06754 (127.8.88.130:42967), 9127714d0c1940198ed545e9565ef40f (127.8.88.129:39089)
I20260928 13:08:55.864733 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.860958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:55867 (local address 127.8.88.130:42967)
0928 13:08:55.861008 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.861011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.861023 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:08:55.861095 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.861096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.861109 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.861177 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:55.861354 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.862137 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.863698 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.864279 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.864366 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.864376 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.864614 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.864664 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 13:08:55.864733 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.861068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:47013 (local address 127.8.88.129:39089)
0928 13:08:55.861236 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.861238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.861245 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:08:55.861270 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.861270 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.861280 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.861356 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:55.861510 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.862191 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.863683 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.864312 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.864374 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.864383 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.864614 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.864657 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260928 13:08:55.864759 16621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.860973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:39089 (local address 127.8.88.131:47013)
0928 13:08:55.861166 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.861176 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.861220 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.861363 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.861364 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.861365 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:55.861419 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.861420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.862263 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.862264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.863551 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.863554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.864384 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.864385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.864521 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.864528 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.864591 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.864644 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":99,"thread_start_us":104,"threads_started":1}
I20260928 13:08:55.864764 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.860821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:42967 (local address 127.8.88.131:55867)
0928 13:08:55.860922 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.860937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.861029 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.861190 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.861192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.861194 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:55.861272 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.861275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.862260 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.862261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.863554 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.863555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.864384 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.864385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.864521 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.864527 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.864591 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.864644 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":9}
I20260928 13:08:55.865235 16251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71d05209d554e6c8d0d71de3b493f5c" candidate_uuid: "811f434ce8b9435c924097885cbe163e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9127714d0c1940198ed545e9565ef40f" is_pre_election: true
I20260928 13:08:55.865357 16398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71d05209d554e6c8d0d71de3b493f5c" candidate_uuid: "811f434ce8b9435c924097885cbe163e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3153063587854129a678e89dd6c06754" is_pre_election: true
I20260928 13:08:55.865422 16251 raft_consensus.cc:2454] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 811f434ce8b9435c924097885cbe163e in term 0.
I20260928 13:08:55.865661 16398 raft_consensus.cc:2454] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 811f434ce8b9435c924097885cbe163e in term 0.
I20260928 13:08:55.865760 16472 leader_election.cc:302] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 811f434ce8b9435c924097885cbe163e, 9127714d0c1940198ed545e9565ef40f; no voters:
I20260928 13:08:55.865917 16619 raft_consensus.cc:2781] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:55.865981 16619 raft_consensus.cc:482] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:55.866008 16619 raft_consensus.cc:3037] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:55.866989 16619 raft_consensus.cc:504] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.867147 16619 leader_election.cc:288] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [CANDIDATE]: Term 1 election: Requested vote from peers 3153063587854129a678e89dd6c06754 (127.8.88.130:42967), 9127714d0c1940198ed545e9565ef40f (127.8.88.129:39089)
I20260928 13:08:55.867309 16398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71d05209d554e6c8d0d71de3b493f5c" candidate_uuid: "811f434ce8b9435c924097885cbe163e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3153063587854129a678e89dd6c06754"
I20260928 13:08:55.867358 16251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71d05209d554e6c8d0d71de3b493f5c" candidate_uuid: "811f434ce8b9435c924097885cbe163e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9127714d0c1940198ed545e9565ef40f"
I20260928 13:08:55.867409 16398 raft_consensus.cc:3037] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:55.867434 16251 raft_consensus.cc:3037] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:55.868276 16251 raft_consensus.cc:2454] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 811f434ce8b9435c924097885cbe163e in term 1.
I20260928 13:08:55.868276 16398 raft_consensus.cc:2454] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 811f434ce8b9435c924097885cbe163e in term 1.
I20260928 13:08:55.868554 16472 leader_election.cc:302] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 811f434ce8b9435c924097885cbe163e, 9127714d0c1940198ed545e9565ef40f; no voters:
I20260928 13:08:55.868660 16619 raft_consensus.cc:2781] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:55.868814 16619 raft_consensus.cc:686] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [term 1 LEADER]: Becoming Leader. State: Replica: 811f434ce8b9435c924097885cbe163e, State: Running, Role: LEADER
I20260928 13:08:55.868948 16619 consensus_queue.cc:237] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.869853 16101 catalog_manager.cc:5495] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e reported cstate change: term changed from 0 to 1, leader changed from <none> to 811f434ce8b9435c924097885cbe163e (127.8.88.131). New cstate: current_term: 1 leader_uuid: "811f434ce8b9435c924097885cbe163e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } health_report { overall_health: HEALTHY } } }
I20260928 13:08:55.887059 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.884196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.8.88.131:39755)
0928 13:08:55.884240 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.884242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.884252 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:08:55.884299 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.884299 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.884319 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.884476 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:55.884646 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.885255 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.886101 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.886295 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.886391 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.886593 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:55.886729 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:55.886881 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.886969 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.886992 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:08:55.889322 16251 raft_consensus.cc:1261] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Refusing update from remote peer 811f434ce8b9435c924097885cbe163e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:55.889369 16398 raft_consensus.cc:1261] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Refusing update from remote peer 811f434ce8b9435c924097885cbe163e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:55.889751 16619 consensus_queue.cc:1035] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:55.889897 16622 consensus_queue.cc:1035] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:55.892828 16625 mvcc.cc:205] Tried to move back new op lower bound from 7334301433399222272 to 7334301433320042496. Current Snapshot: MvccSnapshot[applied={T|T < 7334301433399222272}]
I20260928 13:08:55.893056 16626 mvcc.cc:205] Tried to move back new op lower bound from 7334301433399222272 to 7334301433320042496. Current Snapshot: MvccSnapshot[applied={T|T < 7334301433399222272}]
I20260928 13:08:55.895800 16100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:55.895923 16100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:55.896932 16100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982: Sending DeleteTablet for 3 replicas of tablet f71d05209d554e6c8d0d71de3b493f5c
I20260928 13:08:55.897310 16378 tablet_service.cc:1514] Processing DeleteTablet for tablet f71d05209d554e6c8d0d71de3b493f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:55 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:40224
I20260928 13:08:55.897369 16234 tablet_service.cc:1514] Processing DeleteTablet for tablet f71d05209d554e6c8d0d71de3b493f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:55 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:49484
I20260928 13:08:55.897432 16518 tablet_service.cc:1514] Processing DeleteTablet for tablet f71d05209d554e6c8d0d71de3b493f5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:55 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:50458
I20260928 13:08:55.897755 16636 tablet_replica.cc:330] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e: stopping tablet replica
I20260928 13:08:55.897744 16100 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 13:08:55.897842 16634 tablet_replica.cc:330] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754: stopping tablet replica
I20260928 13:08:55.897862 16636 raft_consensus.cc:2229] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [term 1 LEADER]: Raft consensus shutting down.
W20260928 13:08:55.897900 16100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:55.897949 16634 raft_consensus.cc:2229] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:55.897992 16636 raft_consensus.cc:2258] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:55.898080 16634 raft_consensus.cc:2258] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:55.898633 16635 tablet_replica.cc:330] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f: stopping tablet replica
I20260928 13:08:55.898759 16635 raft_consensus.cc:2229] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:55.898886 16635 raft_consensus.cc:2258] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:55.899222 16636 ts_tablet_manager.cc:1905] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:55.899650 16518 tablet_service.cc:1467] Processing CreateTablet for tablet c6ed45c5c0754582a6e0cdbc03203e62 (DEFAULT_TABLE table=test-table [id=c2df1fc355904e45a5dbc4394c38eef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:55.899711 16234 tablet_service.cc:1467] Processing CreateTablet for tablet c6ed45c5c0754582a6e0cdbc03203e62 (DEFAULT_TABLE table=test-table [id=c2df1fc355904e45a5dbc4394c38eef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:55.899765 16518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ed45c5c0754582a6e0cdbc03203e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:55.899811 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ed45c5c0754582a6e0cdbc03203e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:55.899968 16634 ts_tablet_manager.cc:1905] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:55.901185 16613 tablet_bootstrap.cc:492] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e: Bootstrap starting.
I20260928 13:08:55.901177 16614 tablet_bootstrap.cc:492] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f: Bootstrap starting.
I20260928 13:08:55.901620 16614 tablet_bootstrap.cc:654] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:55.901620 16613 tablet_bootstrap.cc:654] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:55.902148 16378 tablet_service.cc:1467] Processing CreateTablet for tablet c6ed45c5c0754582a6e0cdbc03203e62 (DEFAULT_TABLE table=test-table [id=c2df1fc355904e45a5dbc4394c38eef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:55.902262 16378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ed45c5c0754582a6e0cdbc03203e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:55.903194 16613 tablet_bootstrap.cc:492] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e: No bootstrap required, opened a new log
I20260928 13:08:55.903241 16613 ts_tablet_manager.cc:1397] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:55.903401 16613 raft_consensus.cc:348] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.903481 16613 raft_consensus.cc:374] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:55.903508 16613 raft_consensus.cc:729] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811f434ce8b9435c924097885cbe163e, State: Initialized, Role: FOLLOWER
I20260928 13:08:55.903563 16613 consensus_queue.cc:260] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.903658 16613 ts_tablet_manager.cc:1428] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:55.903733 16612 tablet_bootstrap.cc:492] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754: Bootstrap starting.
I20260928 13:08:55.904201 16612 tablet_bootstrap.cc:654] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:55.904469 16634 ts_tablet_manager.cc:1918] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:55.904560 16634 log.cc:1196] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/f71d05209d554e6c8d0d71de3b493f5c
I20260928 13:08:55.904820 16614 tablet_bootstrap.cc:492] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f: No bootstrap required, opened a new log
I20260928 13:08:55.904844 16634 ts_tablet_manager.cc:1939] T f71d05209d554e6c8d0d71de3b493f5c P 3153063587854129a678e89dd6c06754: Deleting consensus metadata
I20260928 13:08:55.904857 16636 ts_tablet_manager.cc:1918] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:55.904907 16614 ts_tablet_manager.cc:1397] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 13:08:55.904949 16636 log.cc:1196] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/f71d05209d554e6c8d0d71de3b493f5c
I20260928 13:08:55.905082 16614 raft_consensus.cc:348] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.905161 16614 raft_consensus.cc:374] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:55.905189 16614 raft_consensus.cc:729] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9127714d0c1940198ed545e9565ef40f, State: Initialized, Role: FOLLOWER
I20260928 13:08:55.905221 16636 ts_tablet_manager.cc:1939] T f71d05209d554e6c8d0d71de3b493f5c P 811f434ce8b9435c924097885cbe163e: Deleting consensus metadata
I20260928 13:08:55.905238 16614 consensus_queue.cc:260] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.905344 16088 catalog_manager.cc:4841] TS 3153063587854129a678e89dd6c06754 (127.8.88.130:42967): tablet f71d05209d554e6c8d0d71de3b493f5c (table test-table [id=e31ecdb76fb64ca6a4b3a50afbadf9eb]) successfully deleted
I20260928 13:08:55.905385 16614 ts_tablet_manager.cc:1428] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:55.905627 16087 catalog_manager.cc:4841] TS 811f434ce8b9435c924097885cbe163e (127.8.88.131:39755): tablet f71d05209d554e6c8d0d71de3b493f5c (table test-table [id=e31ecdb76fb64ca6a4b3a50afbadf9eb]) successfully deleted
I20260928 13:08:55.904820 16635 ts_tablet_manager.cc:1905] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:55.907624 16612 tablet_bootstrap.cc:492] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754: No bootstrap required, opened a new log
I20260928 13:08:55.907675 16635 ts_tablet_manager.cc:1918] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:55.907696 16612 ts_tablet_manager.cc:1397] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 13:08:55.907750 16635 log.cc:1196] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/wal/wals/f71d05209d554e6c8d0d71de3b493f5c
I20260928 13:08:55.907876 16612 raft_consensus.cc:348] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.907965 16612 raft_consensus.cc:374] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:55.907989 16612 raft_consensus.cc:729] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3153063587854129a678e89dd6c06754, State: Initialized, Role: FOLLOWER
I20260928 13:08:55.908056 16612 consensus_queue.cc:260] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.908177 16612 ts_tablet_manager.cc:1428] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:55.908632 16635 ts_tablet_manager.cc:1939] T f71d05209d554e6c8d0d71de3b493f5c P 9127714d0c1940198ed545e9565ef40f: Deleting consensus metadata
I20260928 13:08:55.909210 16088 catalog_manager.cc:4841] TS 9127714d0c1940198ed545e9565ef40f (127.8.88.129:39089): tablet f71d05209d554e6c8d0d71de3b493f5c (table test-table [id=e31ecdb76fb64ca6a4b3a50afbadf9eb]) successfully deleted
I20260928 13:08:55.938517 16620 raft_consensus.cc:482] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:55.938638 16620 raft_consensus.cc:504] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.939015 16620 leader_election.cc:288] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3153063587854129a678e89dd6c06754 (127.8.88.130:42967), 811f434ce8b9435c924097885cbe163e (127.8.88.131:39755)
I20260928 13:08:55.942173 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.939307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:60845 (local address 127.8.88.131:39755)
0928 13:08:55.939354 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.939356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.939420 (+ 64us) server_negotiation.cc:407] Connection header received
0928 13:08:55.939578 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.939580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.939590 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.939640 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:55.939807 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.940314 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.941424 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.941788 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.941889 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.941900 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.942053 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.942099 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 13:08:55.942186 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.939193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:39755 (local address 127.8.88.129:60845)
0928 13:08:55.939346 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.939358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.939486 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.939643 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.939644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.939645 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:55.939726 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.939728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.940414 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.940415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.941327 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.941329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.941891 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.941892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.941978 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.941983 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.942044 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.942090 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260928 13:08:55.942487 16538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed45c5c0754582a6e0cdbc03203e62" candidate_uuid: "9127714d0c1940198ed545e9565ef40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "811f434ce8b9435c924097885cbe163e" is_pre_election: true
I20260928 13:08:55.942628 16538 raft_consensus.cc:2454] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9127714d0c1940198ed545e9565ef40f in term 0.
I20260928 13:08:55.942842 16191 leader_election.cc:302] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 811f434ce8b9435c924097885cbe163e, 9127714d0c1940198ed545e9565ef40f; no voters:
I20260928 13:08:55.942978 16620 raft_consensus.cc:2781] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:55.943005 16620 raft_consensus.cc:482] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:55.943023 16620 raft_consensus.cc:3037] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:55.943020 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.939395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:35777 (local address 127.8.88.130:42967)
0928 13:08:55.939435 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.939437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.939447 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:08:55.939541 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.939542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.939552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.939604 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:08:55.939816 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.940572 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.942035 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.942584 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.942665 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.942675 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.942878 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.942939 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:08:55.943387 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.939193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:42967 (local address 127.8.88.129:35777)
0928 13:08:55.939402 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.939412 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.939461 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.939659 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.939660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.939662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:08:55.939736 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.939738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.940676 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.940677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.941864 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.941868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.942697 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.942699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.942775 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.942779 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.943171 (+ 392us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.943225 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260928 13:08:55.943655 16398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed45c5c0754582a6e0cdbc03203e62" candidate_uuid: "9127714d0c1940198ed545e9565ef40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3153063587854129a678e89dd6c06754" is_pre_election: true
I20260928 13:08:55.943773 16398 raft_consensus.cc:2454] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9127714d0c1940198ed545e9565ef40f in term 0.
I20260928 13:08:55.943949 16620 raft_consensus.cc:504] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.944108 16620 leader_election.cc:288] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [CANDIDATE]: Term 1 election: Requested vote from peers 3153063587854129a678e89dd6c06754 (127.8.88.130:42967), 811f434ce8b9435c924097885cbe163e (127.8.88.131:39755)
I20260928 13:08:55.944303 16538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed45c5c0754582a6e0cdbc03203e62" candidate_uuid: "9127714d0c1940198ed545e9565ef40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "811f434ce8b9435c924097885cbe163e"
I20260928 13:08:55.944334 16398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ed45c5c0754582a6e0cdbc03203e62" candidate_uuid: "9127714d0c1940198ed545e9565ef40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3153063587854129a678e89dd6c06754"
I20260928 13:08:55.944392 16538 raft_consensus.cc:3037] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:55.944404 16398 raft_consensus.cc:3037] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:55.945134 16538 raft_consensus.cc:2454] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9127714d0c1940198ed545e9565ef40f in term 1.
I20260928 13:08:55.945209 16398 raft_consensus.cc:2454] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9127714d0c1940198ed545e9565ef40f in term 1.
I20260928 13:08:55.945400 16192 leader_election.cc:302] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3153063587854129a678e89dd6c06754, 9127714d0c1940198ed545e9565ef40f; no voters:
I20260928 13:08:55.945489 16620 raft_consensus.cc:2781] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:55.945627 16620 raft_consensus.cc:686] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [term 1 LEADER]: Becoming Leader. State: Replica: 9127714d0c1940198ed545e9565ef40f, State: Running, Role: LEADER
I20260928 13:08:55.945695 16620 consensus_queue.cc:237] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.946470 16101 catalog_manager.cc:5495] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9127714d0c1940198ed545e9565ef40f (127.8.88.129). New cstate: current_term: 1 leader_uuid: "9127714d0c1940198ed545e9565ef40f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } health_report { overall_health: UNKNOWN } } }
Sep 28 13:08:55 dist-test-slave-p1lc krb5kdc[16056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600935, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.190@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
I20260928 13:08:55.977555 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.969771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.8.88.190:46851)
0928 13:08:55.969824 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:55.969827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:55.970104 (+ 277us) server_negotiation.cc:407] Connection header received
0928 13:08:55.970136 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:55.970137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:55.970150 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:55.970205 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:55.970667 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.971428 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.972281 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.972550 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.972635 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.974832 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:55.974834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:55.974834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:55.974842 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:55.976261 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.976689 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.976690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.976691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.976743 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:55.976980 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:55.976981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:55.976982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:55.977095 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:55.977173 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:55.977330 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 13:08:55.977446 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:08:55.977653 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:55.969719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46851 (local address 127.0.0.1:52890)
0928 13:08:55.969981 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:55.969991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:55.970060 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:55.970218 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:55.970219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:55.970515 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:55.970577 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.970579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.971518 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.971519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.972135 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:55.972137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:55.972638 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:55.972639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:55.972693 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:55.973286 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:55.973290 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:55.974741 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:55.976392 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.976399 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.976402 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.976601 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.976817 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:55.976818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:55.976819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:55.976930 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:55.977164 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:55.977171 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:55.977218 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 13:08:55.977306 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 13:08:55.977529 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":109,"thread_start_us":171,"threads_started":1}
I20260928 13:08:55.980466 16101 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:52890:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 13:08:55.980731 16101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:55.982344 16234 tablet_service.cc:1467] Processing CreateTablet for tablet 21c285951d8b4f9bbf34343870b104bb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e226c89bcfc461fa8a168e71fb4e0e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 13:08:55.982344 16518 tablet_service.cc:1467] Processing CreateTablet for tablet 21c285951d8b4f9bbf34343870b104bb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e226c89bcfc461fa8a168e71fb4e0e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 13:08:55.982354 16378 tablet_service.cc:1467] Processing CreateTablet for tablet 21c285951d8b4f9bbf34343870b104bb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e226c89bcfc461fa8a168e71fb4e0e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 13:08:55.982508 16378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c285951d8b4f9bbf34343870b104bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:55.982508 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c285951d8b4f9bbf34343870b104bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:55.982508 16518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c285951d8b4f9bbf34343870b104bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:55.984030 16614 tablet_bootstrap.cc:492] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f: Bootstrap starting.
I20260928 13:08:55.984174 16612 tablet_bootstrap.cc:492] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754: Bootstrap starting.
I20260928 13:08:55.984408 16614 tablet_bootstrap.cc:654] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:55.984566 16612 tablet_bootstrap.cc:654] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:55.984752 16613 tablet_bootstrap.cc:492] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e: Bootstrap starting.
I20260928 13:08:55.985183 16613 tablet_bootstrap.cc:654] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:55.985692 16612 tablet_bootstrap.cc:492] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754: No bootstrap required, opened a new log
I20260928 13:08:55.985747 16612 ts_tablet_manager.cc:1397] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:55.985906 16612 raft_consensus.cc:348] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.985992 16612 raft_consensus.cc:374] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:55.986019 16612 raft_consensus.cc:729] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3153063587854129a678e89dd6c06754, State: Initialized, Role: FOLLOWER
I20260928 13:08:55.986066 16612 consensus_queue.cc:260] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.986095 16614 tablet_bootstrap.cc:492] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f: No bootstrap required, opened a new log
I20260928 13:08:55.986136 16614 ts_tablet_manager.cc:1397] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 13:08:55.986176 16612 ts_tablet_manager.cc:1428] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:55.986275 16614 raft_consensus.cc:348] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.986342 16614 raft_consensus.cc:374] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:55.986362 16614 raft_consensus.cc:729] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9127714d0c1940198ed545e9565ef40f, State: Initialized, Role: FOLLOWER
I20260928 13:08:55.986357 16618 tablet_replica.cc:439] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } } }
I20260928 13:08:55.986408 16614 consensus_queue.cc:260] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.986440 16618 tablet_replica.cc:442] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:55.986502 16614 ts_tablet_manager.cc:1428] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:55.986552 16620 tablet_replica.cc:439] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } } }
I20260928 13:08:55.986640 16620 tablet_replica.cc:442] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:55.986697 16613 tablet_bootstrap.cc:492] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e: No bootstrap required, opened a new log
I20260928 13:08:55.986742 16613 ts_tablet_manager.cc:1397] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:55.986897 16613 raft_consensus.cc:348] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.986984 16613 raft_consensus.cc:374] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:55.987005 16613 raft_consensus.cc:729] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811f434ce8b9435c924097885cbe163e, State: Initialized, Role: FOLLOWER
I20260928 13:08:55.987051 16613 consensus_queue.cc:260] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:55.987143 16613 ts_tablet_manager.cc:1428] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:55.987248 16619 tablet_replica.cc:439] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } } }
I20260928 13:08:55.987303 16619 tablet_replica.cc:442] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 13:08:56.147984 16585 tablet.cc:2369] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:08:56.289201 16620 raft_consensus.cc:482] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:56.289306 16620 raft_consensus.cc:504] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.289494 16620 leader_election.cc:288] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3153063587854129a678e89dd6c06754 (127.8.88.130:42967), 811f434ce8b9435c924097885cbe163e (127.8.88.131:39755)
I20260928 13:08:56.289740 16398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c285951d8b4f9bbf34343870b104bb" candidate_uuid: "9127714d0c1940198ed545e9565ef40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3153063587854129a678e89dd6c06754" is_pre_election: true
I20260928 13:08:56.289851 16398 raft_consensus.cc:2454] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9127714d0c1940198ed545e9565ef40f in term 0.
I20260928 13:08:56.289755 16538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c285951d8b4f9bbf34343870b104bb" candidate_uuid: "9127714d0c1940198ed545e9565ef40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "811f434ce8b9435c924097885cbe163e" is_pre_election: true
I20260928 13:08:56.289906 16538 raft_consensus.cc:2454] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9127714d0c1940198ed545e9565ef40f in term 0.
I20260928 13:08:56.290009 16192 leader_election.cc:302] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3153063587854129a678e89dd6c06754, 9127714d0c1940198ed545e9565ef40f; no voters:
I20260928 13:08:56.290215 16620 raft_consensus.cc:2781] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:56.290258 16620 raft_consensus.cc:482] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:56.290272 16620 raft_consensus.cc:3037] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:56.290936 16620 raft_consensus.cc:504] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.291108 16620 leader_election.cc:288] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [CANDIDATE]: Term 1 election: Requested vote from peers 3153063587854129a678e89dd6c06754 (127.8.88.130:42967), 811f434ce8b9435c924097885cbe163e (127.8.88.131:39755)
I20260928 13:08:56.291244 16398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c285951d8b4f9bbf34343870b104bb" candidate_uuid: "9127714d0c1940198ed545e9565ef40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3153063587854129a678e89dd6c06754"
I20260928 13:08:56.291289 16538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c285951d8b4f9bbf34343870b104bb" candidate_uuid: "9127714d0c1940198ed545e9565ef40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "811f434ce8b9435c924097885cbe163e"
I20260928 13:08:56.291343 16398 raft_consensus.cc:3037] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:56.291352 16538 raft_consensus.cc:3037] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:56.292078 16398 raft_consensus.cc:2454] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9127714d0c1940198ed545e9565ef40f in term 1.
I20260928 13:08:56.292119 16538 raft_consensus.cc:2454] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9127714d0c1940198ed545e9565ef40f in term 1.
I20260928 13:08:56.292268 16192 leader_election.cc:302] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3153063587854129a678e89dd6c06754, 9127714d0c1940198ed545e9565ef40f; no voters:
I20260928 13:08:56.292395 16620 raft_consensus.cc:2781] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:56.292445 16620 raft_consensus.cc:686] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [term 1 LEADER]: Becoming Leader. State: Replica: 9127714d0c1940198ed545e9565ef40f, State: Running, Role: LEADER
I20260928 13:08:56.292486 16620 consensus_queue.cc:237] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.292848 16642 tablet_replica.cc:439] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f: TxnStatusTablet state changed. Reason: New leader 9127714d0c1940198ed545e9565ef40f. Latest consensus state: current_term: 1 leader_uuid: "9127714d0c1940198ed545e9565ef40f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } } }
I20260928 13:08:56.293208 16642 tablet_replica.cc:442] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f: This TxnStatusTablet replica's current role is: LEADER
I20260928 13:08:56.293310 16101 catalog_manager.cc:5495] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9127714d0c1940198ed545e9565ef40f (127.8.88.129). New cstate: current_term: 1 leader_uuid: "9127714d0c1940198ed545e9565ef40f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } health_report { overall_health: UNKNOWN } } }
I20260928 13:08:56.293468 16654 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 13:08:56.293586 16654 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 13:08:56.349171 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:56.346380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:39089 (local address 127.0.0.1:49500)
0928 13:08:56.346537 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:56.346575 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:56.346636 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:56.346854 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:56.346855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:56.346856 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 13:08:56.346912 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:56.346914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.347581 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.347582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:56.348157 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:56.348159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.348548 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.348549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:56.348616 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:56.348630 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 13:08:56.348987 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 13:08:56.348998 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 13:08:56.349044 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 13:08:56.349097 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 13:08:56.349171 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:56.346537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.8.88.129:39089)
0928 13:08:56.346607 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:56.346609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:56.346618 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:08:56.346738 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:56.346740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:56.346768 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:56.346849 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:56.346996 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.347480 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.348248 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.348447 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.348531 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:56.348728 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:56.348835 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:56.349001 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:56.349059 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 13:08:56.349082 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 13:08:56.350347 16398 raft_consensus.cc:1261] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Refusing update from remote peer 9127714d0c1940198ed545e9565ef40f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:56.350452 16538 raft_consensus.cc:1261] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Refusing update from remote peer 9127714d0c1940198ed545e9565ef40f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:56.350607 16620 consensus_queue.cc:1035] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:56.350711 16642 consensus_queue.cc:1035] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f [LEADER]: Connected to new peer: Peer: permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:56.351267 16618 tablet_replica.cc:439] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754: TxnStatusTablet state changed. Reason: New leader 9127714d0c1940198ed545e9565ef40f. Latest consensus state: current_term: 1 leader_uuid: "9127714d0c1940198ed545e9565ef40f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } } }
I20260928 13:08:56.351379 16618 tablet_replica.cc:442] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:56.351415 16619 tablet_replica.cc:439] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e: TxnStatusTablet state changed. Reason: New leader 9127714d0c1940198ed545e9565ef40f. Latest consensus state: current_term: 1 leader_uuid: "9127714d0c1940198ed545e9565ef40f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } } }
I20260928 13:08:56.351500 16619 tablet_replica.cc:442] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:56.352347 16642 tablet_replica.cc:439] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9127714d0c1940198ed545e9565ef40f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } } }
I20260928 13:08:56.352434 16642 tablet_replica.cc:442] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f: This TxnStatusTablet replica's current role is: LEADER
I20260928 13:08:56.352609 16619 tablet_replica.cc:439] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9127714d0c1940198ed545e9565ef40f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } } }
I20260928 13:08:56.352689 16619 tablet_replica.cc:442] T 21c285951d8b4f9bbf34343870b104bb P 811f434ce8b9435c924097885cbe163e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:56.352978 16620 tablet_replica.cc:439] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9127714d0c1940198ed545e9565ef40f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } } }
I20260928 13:08:56.353070 16620 tablet_replica.cc:442] T 21c285951d8b4f9bbf34343870b104bb P 9127714d0c1940198ed545e9565ef40f: This TxnStatusTablet replica's current role is: LEADER
I20260928 13:08:56.353324 16618 tablet_replica.cc:439] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9127714d0c1940198ed545e9565ef40f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } } }
I20260928 13:08:56.353400 16618 tablet_replica.cc:442] T 21c285951d8b4f9bbf34343870b104bb P 3153063587854129a678e89dd6c06754: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 13:08:56.356786 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:56.354514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49506 (local address 127.8.88.129:39089)
0928 13:08:56.354551 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:56.354554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:56.354593 (+ 39us) server_negotiation.cc:407] Connection header received
0928 13:08:56.354661 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:56.354661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:56.354671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:56.354745 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:56.354889 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.355353 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.356108 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.356314 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.356392 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:56.356534 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:56.356566 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:56.356611 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:56.356679 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 13:08:56.356701 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:08:56.362437 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:56.359617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:49711 (local address 127.8.88.129:39089)
0928 13:08:56.359664 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:56.359666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:56.359764 (+ 98us) server_negotiation.cc:407] Connection header received
0928 13:08:56.359823 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:56.359823 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:56.359834 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:56.359902 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:56.360032 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.360725 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.361568 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.361834 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.361915 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:56.362155 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:56.362189 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:56.362255 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:56.362311 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 13:08:56.362337 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:08:56.362564 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:56.359550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:39089 (local address 127.8.88.129:49711)
0928 13:08:56.359684 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:56.359695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:56.359780 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:56.359908 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:56.359909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:56.359910 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 13:08:56.359970 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:56.359971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.360866 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.360868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:56.361474 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:56.361476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.361975 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.361977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:56.362045 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:56.362059 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 13:08:56.362266 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 13:08:56.362268 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 13:08:56.362305 (+ 37us) client_negotiation.cc:240] Negotiation successful
0928 13:08:56.362345 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260928 13:08:56.365190 16538 raft_consensus.cc:1261] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Refusing update from remote peer 9127714d0c1940198ed545e9565ef40f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:56.365228 16398 raft_consensus.cc:1261] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Refusing update from remote peer 9127714d0c1940198ed545e9565ef40f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:56.365451 16659 consensus_queue.cc:1035] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [LEADER]: Connected to new peer: Peer: permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:56.365552 16620 consensus_queue.cc:1035] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:56.367183 16627 mvcc.cc:205] Tried to move back new op lower bound from 7334301435349831680 to 7334301433634263040. Current Snapshot: MvccSnapshot[applied={T|T < 7334301435349831680}]
I20260928 13:08:56.380261 16100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:56.380414 16100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:56.381147 16100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982: Sending DeleteTablet for 3 replicas of tablet c6ed45c5c0754582a6e0cdbc03203e62
I20260928 13:08:56.381428 16518 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ed45c5c0754582a6e0cdbc03203e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:56 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:50458
I20260928 13:08:56.381428 16378 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ed45c5c0754582a6e0cdbc03203e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:56 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:40224
I20260928 13:08:56.381529 16636 tablet_replica.cc:330] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e: stopping tablet replica
I20260928 13:08:56.381547 16634 tablet_replica.cc:330] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754: stopping tablet replica
I20260928 13:08:56.381585 16636 raft_consensus.cc:2229] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:56.381620 16636 raft_consensus.cc:2258] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:56.381628 16634 raft_consensus.cc:2229] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:56.381678 16634 raft_consensus.cc:2258] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:56.381734 16234 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ed45c5c0754582a6e0cdbc03203e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:56 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:49484
I20260928 13:08:56.381819 16635 tablet_replica.cc:330] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f: stopping tablet replica
I20260928 13:08:56.381870 16635 raft_consensus.cc:2229] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:08:56.381958 16635 raft_consensus.cc:2258] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:56.382083 16634 ts_tablet_manager.cc:1905] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:56.382503 16635 ts_tablet_manager.cc:1905] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:56.383960 16634 ts_tablet_manager.cc:1918] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:56.384017 16634 log.cc:1196] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/c6ed45c5c0754582a6e0cdbc03203e62
I20260928 13:08:56.384251 16634 ts_tablet_manager.cc:1939] T c6ed45c5c0754582a6e0cdbc03203e62 P 3153063587854129a678e89dd6c06754: Deleting consensus metadata
I20260928 13:08:56.384447 16635 ts_tablet_manager.cc:1918] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:56.384513 16635 log.cc:1196] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/wal/wals/c6ed45c5c0754582a6e0cdbc03203e62
I20260928 13:08:56.384769 16635 ts_tablet_manager.cc:1939] T c6ed45c5c0754582a6e0cdbc03203e62 P 9127714d0c1940198ed545e9565ef40f: Deleting consensus metadata
I20260928 13:08:56.385317 16088 catalog_manager.cc:4841] TS 3153063587854129a678e89dd6c06754 (127.8.88.130:42967): tablet c6ed45c5c0754582a6e0cdbc03203e62 (table test-table [id=c2df1fc355904e45a5dbc4394c38eef5]) successfully deleted
I20260928 13:08:56.385442 16636 ts_tablet_manager.cc:1905] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:56.385514 16088 catalog_manager.cc:4841] TS 9127714d0c1940198ed545e9565ef40f (127.8.88.129:39089): tablet c6ed45c5c0754582a6e0cdbc03203e62 (table test-table [id=c2df1fc355904e45a5dbc4394c38eef5]) successfully deleted
I20260928 13:08:56.387149 16636 ts_tablet_manager.cc:1918] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:56.387215 16636 log.cc:1196] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/c6ed45c5c0754582a6e0cdbc03203e62
I20260928 13:08:56.387446 16636 ts_tablet_manager.cc:1939] T c6ed45c5c0754582a6e0cdbc03203e62 P 811f434ce8b9435c924097885cbe163e: Deleting consensus metadata
I20260928 13:08:56.387887 16087 catalog_manager.cc:4841] TS 811f434ce8b9435c924097885cbe163e (127.8.88.131:39755): tablet c6ed45c5c0754582a6e0cdbc03203e62 (table test-table [id=c2df1fc355904e45a5dbc4394c38eef5]) successfully deleted
Sep 28 13:08:56 dist-test-slave-p1lc krb5kdc[16056](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 13:08:56 dist-test-slave-p1lc krb5kdc[16056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:08:56 dist-test-slave-p1lc krb5kdc[16056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
W20260928 13:08:56.407475 16151 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:08:56.401311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.8.88.190:46851)
0928 13:08:56.401395 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:56.401398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:56.401518 (+ 120us) server_negotiation.cc:407] Connection header received
0928 13:08:56.401610 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:56.401611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:56.401623 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:56.401696 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:56.402464 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.403104 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.403919 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.404207 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.404316 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:56.407384 (+ 3068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52894: BlockingRecv error: recv got EOF from 127.0.0.1:52894 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 13:08:56 dist-test-slave-p1lc krb5kdc[16056](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM
I20260928 13:08:56.417104 16151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:56.408453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52902 (local address 127.8.88.190:46851)
0928 13:08:56.408522 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:56.408524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:56.408595 (+ 71us) server_negotiation.cc:407] Connection header received
0928 13:08:56.408650 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:56.408652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:56.408663 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:56.408739 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:56.409306 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.410065 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.410785 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.411075 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.411155 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:56.413769 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:56.413771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:56.413772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:56.413780 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:56.415634 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:56.416198 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:56.416200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:56.416200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:56.416269 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:56.416480 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:56.416481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:56.416481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:56.416627 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:56.416728 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:56.416916 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 13:08:56.417004 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 13:08:56.418977 16100 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:08:56.419179 16100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:56.420727 16518 tablet_service.cc:1467] Processing CreateTablet for tablet 284cace0beb440b28af9f89e78c38af3 (DEFAULT_TABLE table=test-table [id=c0c15b39b0694a1f9e0cc4b39c90e322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:56.420719 16234 tablet_service.cc:1467] Processing CreateTablet for tablet 284cace0beb440b28af9f89e78c38af3 (DEFAULT_TABLE table=test-table [id=c0c15b39b0694a1f9e0cc4b39c90e322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:56.420785 16378 tablet_service.cc:1467] Processing CreateTablet for tablet 284cace0beb440b28af9f89e78c38af3 (DEFAULT_TABLE table=test-table [id=c0c15b39b0694a1f9e0cc4b39c90e322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:56.420861 16518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 284cace0beb440b28af9f89e78c38af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:56.420861 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 284cace0beb440b28af9f89e78c38af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:56.420979 16378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 284cace0beb440b28af9f89e78c38af3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:56.422271 16612 tablet_bootstrap.cc:492] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754: Bootstrap starting.
I20260928 13:08:56.422398 16613 tablet_bootstrap.cc:492] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e: Bootstrap starting.
I20260928 13:08:56.422688 16613 tablet_bootstrap.cc:654] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:56.422691 16612 tablet_bootstrap.cc:654] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:56.423013 16614 tablet_bootstrap.cc:492] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f: Bootstrap starting.
I20260928 13:08:56.423362 16614 tablet_bootstrap.cc:654] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:56.423569 16612 tablet_bootstrap.cc:492] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754: No bootstrap required, opened a new log
I20260928 13:08:56.423616 16612 ts_tablet_manager.cc:1397] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:56.423811 16612 raft_consensus.cc:348] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.423887 16612 raft_consensus.cc:374] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:56.423904 16612 raft_consensus.cc:729] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3153063587854129a678e89dd6c06754, State: Initialized, Role: FOLLOWER
I20260928 13:08:56.423947 16612 consensus_queue.cc:260] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.424029 16612 ts_tablet_manager.cc:1428] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:56.424402 16613 tablet_bootstrap.cc:492] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e: No bootstrap required, opened a new log
I20260928 13:08:56.424468 16613 ts_tablet_manager.cc:1397] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:08:56.424593 16613 raft_consensus.cc:348] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.424666 16613 raft_consensus.cc:374] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:56.424698 16613 raft_consensus.cc:729] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811f434ce8b9435c924097885cbe163e, State: Initialized, Role: FOLLOWER
I20260928 13:08:56.424750 16613 consensus_queue.cc:260] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.425057 16614 tablet_bootstrap.cc:492] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f: No bootstrap required, opened a new log
I20260928 13:08:56.425132 16614 ts_tablet_manager.cc:1397] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 13:08:56.425309 16614 raft_consensus.cc:348] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.425380 16614 raft_consensus.cc:374] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:56.425402 16614 raft_consensus.cc:729] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9127714d0c1940198ed545e9565ef40f, State: Initialized, Role: FOLLOWER
I20260928 13:08:56.425447 16614 consensus_queue.cc:260] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.425530 16614 ts_tablet_manager.cc:1428] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 13:08:56.424853 16613 ts_tablet_manager.cc:1428] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:56.474139 16642 raft_consensus.cc:482] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:56.474236 16642 raft_consensus.cc:504] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.474423 16642 leader_election.cc:288] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3153063587854129a678e89dd6c06754 (127.8.88.130:42967), 811f434ce8b9435c924097885cbe163e (127.8.88.131:39755)
I20260928 13:08:56.474607 16398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "284cace0beb440b28af9f89e78c38af3" candidate_uuid: "9127714d0c1940198ed545e9565ef40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3153063587854129a678e89dd6c06754" is_pre_election: true
I20260928 13:08:56.474717 16398 raft_consensus.cc:2454] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9127714d0c1940198ed545e9565ef40f in term 0.
I20260928 13:08:56.474673 16538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "284cace0beb440b28af9f89e78c38af3" candidate_uuid: "9127714d0c1940198ed545e9565ef40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "811f434ce8b9435c924097885cbe163e" is_pre_election: true
I20260928 13:08:56.474776 16538 raft_consensus.cc:2454] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9127714d0c1940198ed545e9565ef40f in term 0.
I20260928 13:08:56.474946 16192 leader_election.cc:302] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3153063587854129a678e89dd6c06754, 9127714d0c1940198ed545e9565ef40f; no voters:
I20260928 13:08:56.475044 16642 raft_consensus.cc:2781] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:56.475068 16642 raft_consensus.cc:482] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:56.475087 16642 raft_consensus.cc:3037] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:56.475760 16642 raft_consensus.cc:504] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.475903 16642 leader_election.cc:288] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [CANDIDATE]: Term 1 election: Requested vote from peers 3153063587854129a678e89dd6c06754 (127.8.88.130:42967), 811f434ce8b9435c924097885cbe163e (127.8.88.131:39755)
I20260928 13:08:56.476089 16398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "284cace0beb440b28af9f89e78c38af3" candidate_uuid: "9127714d0c1940198ed545e9565ef40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3153063587854129a678e89dd6c06754"
I20260928 13:08:56.476176 16398 raft_consensus.cc:3037] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:56.476986 16538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "284cace0beb440b28af9f89e78c38af3" candidate_uuid: "9127714d0c1940198ed545e9565ef40f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "811f434ce8b9435c924097885cbe163e"
I20260928 13:08:56.477035 16398 raft_consensus.cc:2454] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9127714d0c1940198ed545e9565ef40f in term 1.
I20260928 13:08:56.477080 16538 raft_consensus.cc:3037] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:56.477250 16192 leader_election.cc:302] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3153063587854129a678e89dd6c06754, 9127714d0c1940198ed545e9565ef40f; no voters:
I20260928 13:08:56.477340 16642 raft_consensus.cc:2781] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:56.477396 16642 raft_consensus.cc:686] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [term 1 LEADER]: Becoming Leader. State: Replica: 9127714d0c1940198ed545e9565ef40f, State: Running, Role: LEADER
I20260928 13:08:56.477437 16642 consensus_queue.cc:237] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.477990 16538 raft_consensus.cc:2454] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9127714d0c1940198ed545e9565ef40f in term 1.
I20260928 13:08:56.477993 16101 catalog_manager.cc:5495] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9127714d0c1940198ed545e9565ef40f (127.8.88.129). New cstate: current_term: 1 leader_uuid: "9127714d0c1940198ed545e9565ef40f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } health_report { overall_health: UNKNOWN } } }
I20260928 13:08:56.488479 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:56.486040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.8.88.129:39089)
0928 13:08:56.486086 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:56.486088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:56.486097 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:08:56.486154 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:56.486155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:56.486166 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:56.486218 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:56.486389 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.486907 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.487725 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.487939 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.488033 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:56.488204 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:56.488247 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:56.488303 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:56.488364 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 13:08:56.488385 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:08:56.489423 16538 raft_consensus.cc:1261] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Refusing update from remote peer 9127714d0c1940198ed545e9565ef40f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:56.489599 16398 raft_consensus.cc:1261] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Refusing update from remote peer 9127714d0c1940198ed545e9565ef40f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:56.489809 16620 consensus_queue.cc:1035] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [LEADER]: Connected to new peer: Peer: permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:56.489936 16642 consensus_queue.cc:1035] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:56.493034 16101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:56.493145 16101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:56.493857 16101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982: Sending DeleteTablet for 3 replicas of tablet 284cace0beb440b28af9f89e78c38af3
I20260928 13:08:56.494166 16378 tablet_service.cc:1514] Processing DeleteTablet for tablet 284cace0beb440b28af9f89e78c38af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:56 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:40224
I20260928 13:08:56.494176 16234 tablet_service.cc:1514] Processing DeleteTablet for tablet 284cace0beb440b28af9f89e78c38af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:56 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:49484
I20260928 13:08:56.494267 16634 tablet_replica.cc:330] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754: stopping tablet replica
I20260928 13:08:56.494294 16635 tablet_replica.cc:330] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f: stopping tablet replica
I20260928 13:08:56.494313 16634 raft_consensus.cc:2229] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:56.494354 16635 raft_consensus.cc:2229] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:08:56.494386 16634 raft_consensus.cc:2258] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:56.494412 16635 raft_consensus.cc:2258] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:56.494516 16518 tablet_service.cc:1514] Processing DeleteTablet for tablet 284cace0beb440b28af9f89e78c38af3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:56 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:50458
I20260928 13:08:56.494616 16636 tablet_replica.cc:330] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e: stopping tablet replica
I20260928 13:08:56.494576 16101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 13:08:56.494670 16636 raft_consensus.cc:2229] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:56.494705 16636 raft_consensus.cc:2258] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 13:08:56.494705 16101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:08:56.495103 16636 ts_tablet_manager.cc:1905] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:56.495945 16518 tablet_service.cc:1467] Processing CreateTablet for tablet 40dba7cbeb5a4da0af3d08a38b099c0e (DEFAULT_TABLE table=test-table [id=641e8c5db216485fa876285e91a324de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:56.495955 16234 tablet_service.cc:1467] Processing CreateTablet for tablet 40dba7cbeb5a4da0af3d08a38b099c0e (DEFAULT_TABLE table=test-table [id=641e8c5db216485fa876285e91a324de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:56.496068 16234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40dba7cbeb5a4da0af3d08a38b099c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:56.496068 16518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40dba7cbeb5a4da0af3d08a38b099c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:56.496644 16378 tablet_service.cc:1467] Processing CreateTablet for tablet 40dba7cbeb5a4da0af3d08a38b099c0e (DEFAULT_TABLE table=test-table [id=641e8c5db216485fa876285e91a324de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:08:56.496750 16378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40dba7cbeb5a4da0af3d08a38b099c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:56.497457 16614 tablet_bootstrap.cc:492] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f: Bootstrap starting.
I20260928 13:08:56.497804 16614 tablet_bootstrap.cc:654] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:56.498085 16612 tablet_bootstrap.cc:492] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754: Bootstrap starting.
I20260928 13:08:56.498365 16614 tablet_bootstrap.cc:492] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f: No bootstrap required, opened a new log
I20260928 13:08:56.498435 16614 ts_tablet_manager.cc:1397] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:56.498553 16614 raft_consensus.cc:348] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.498634 16614 raft_consensus.cc:374] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:56.498667 16614 raft_consensus.cc:729] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9127714d0c1940198ed545e9565ef40f, State: Initialized, Role: FOLLOWER
I20260928 13:08:56.498721 16612 tablet_bootstrap.cc:654] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:56.498716 16614 consensus_queue.cc:260] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.498809 16614 ts_tablet_manager.cc:1428] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 13:08:56.499482 16612 tablet_bootstrap.cc:492] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754: No bootstrap required, opened a new log
I20260928 13:08:56.499534 16612 ts_tablet_manager.cc:1397] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:56.499691 16612 raft_consensus.cc:348] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.499761 16612 raft_consensus.cc:374] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:56.499782 16612 raft_consensus.cc:729] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3153063587854129a678e89dd6c06754, State: Initialized, Role: FOLLOWER
I20260928 13:08:56.499833 16612 consensus_queue.cc:260] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.499928 16612 ts_tablet_manager.cc:1428] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 13:08:56.500128 16636 ts_tablet_manager.cc:1918] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:56.500186 16636 log.cc:1196] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/284cace0beb440b28af9f89e78c38af3
I20260928 13:08:56.500452 16636 ts_tablet_manager.cc:1939] T 284cace0beb440b28af9f89e78c38af3 P 811f434ce8b9435c924097885cbe163e: Deleting consensus metadata
I20260928 13:08:56.500772 16087 catalog_manager.cc:4841] TS 811f434ce8b9435c924097885cbe163e (127.8.88.131:39755): tablet 284cace0beb440b28af9f89e78c38af3 (table test-table [id=c0c15b39b0694a1f9e0cc4b39c90e322]) successfully deleted
I20260928 13:08:56.501519 16613 tablet_bootstrap.cc:492] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e: Bootstrap starting.
I20260928 13:08:56.501634 16634 ts_tablet_manager.cc:1905] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:56.501907 16613 tablet_bootstrap.cc:654] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:56.502528 16613 tablet_bootstrap.cc:492] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e: No bootstrap required, opened a new log
I20260928 13:08:56.502595 16613 ts_tablet_manager.cc:1397] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 13:08:56.502702 16613 raft_consensus.cc:348] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.502806 16613 raft_consensus.cc:374] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:56.502835 16613 raft_consensus.cc:729] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811f434ce8b9435c924097885cbe163e, State: Initialized, Role: FOLLOWER
I20260928 13:08:56.502876 16613 consensus_queue.cc:260] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.502965 16613 ts_tablet_manager.cc:1428] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 13:08:56.503294 16634 ts_tablet_manager.cc:1918] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:56.503366 16634 log.cc:1196] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/284cace0beb440b28af9f89e78c38af3
I20260928 13:08:56.503369 16635 ts_tablet_manager.cc:1905] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:56.503611 16634 ts_tablet_manager.cc:1939] T 284cace0beb440b28af9f89e78c38af3 P 3153063587854129a678e89dd6c06754: Deleting consensus metadata
I20260928 13:08:56.503959 16088 catalog_manager.cc:4841] TS 3153063587854129a678e89dd6c06754 (127.8.88.130:42967): tablet 284cace0beb440b28af9f89e78c38af3 (table test-table [id=c0c15b39b0694a1f9e0cc4b39c90e322]) successfully deleted
I20260928 13:08:56.504678 16635 ts_tablet_manager.cc:1918] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:08:56.504734 16635 log.cc:1196] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-0/wal/wals/284cace0beb440b28af9f89e78c38af3
I20260928 13:08:56.504988 16635 ts_tablet_manager.cc:1939] T 284cace0beb440b28af9f89e78c38af3 P 9127714d0c1940198ed545e9565ef40f: Deleting consensus metadata
I20260928 13:08:56.505304 16088 catalog_manager.cc:4841] TS 9127714d0c1940198ed545e9565ef40f (127.8.88.129:39089): tablet 284cace0beb440b28af9f89e78c38af3 (table test-table [id=c0c15b39b0694a1f9e0cc4b39c90e322]) successfully deleted
I20260928 13:08:56.626909 16619 raft_consensus.cc:482] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:08:56.627015 16619 raft_consensus.cc:504] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.627170 16619 leader_election.cc:288] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9127714d0c1940198ed545e9565ef40f (127.8.88.129:39089), 3153063587854129a678e89dd6c06754 (127.8.88.130:42967)
I20260928 13:08:56.627576 16251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40dba7cbeb5a4da0af3d08a38b099c0e" candidate_uuid: "811f434ce8b9435c924097885cbe163e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9127714d0c1940198ed545e9565ef40f" is_pre_election: true
I20260928 13:08:56.627576 16398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40dba7cbeb5a4da0af3d08a38b099c0e" candidate_uuid: "811f434ce8b9435c924097885cbe163e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3153063587854129a678e89dd6c06754" is_pre_election: true
I20260928 13:08:56.627701 16251 raft_consensus.cc:2454] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 811f434ce8b9435c924097885cbe163e in term 0.
I20260928 13:08:56.627701 16398 raft_consensus.cc:2454] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 811f434ce8b9435c924097885cbe163e in term 0.
I20260928 13:08:56.627975 16472 leader_election.cc:302] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 811f434ce8b9435c924097885cbe163e, 9127714d0c1940198ed545e9565ef40f; no voters:
I20260928 13:08:56.628099 16619 raft_consensus.cc:2781] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:08:56.628134 16619 raft_consensus.cc:482] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:08:56.628167 16619 raft_consensus.cc:3037] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:56.628824 16619 raft_consensus.cc:504] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.629041 16619 leader_election.cc:288] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [CANDIDATE]: Term 1 election: Requested vote from peers 9127714d0c1940198ed545e9565ef40f (127.8.88.129:39089), 3153063587854129a678e89dd6c06754 (127.8.88.130:42967)
I20260928 13:08:56.629226 16251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40dba7cbeb5a4da0af3d08a38b099c0e" candidate_uuid: "811f434ce8b9435c924097885cbe163e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9127714d0c1940198ed545e9565ef40f"
I20260928 13:08:56.629242 16398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40dba7cbeb5a4da0af3d08a38b099c0e" candidate_uuid: "811f434ce8b9435c924097885cbe163e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3153063587854129a678e89dd6c06754"
I20260928 13:08:56.629310 16251 raft_consensus.cc:3037] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:56.629343 16398 raft_consensus.cc:3037] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:56.630098 16398 raft_consensus.cc:2454] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 811f434ce8b9435c924097885cbe163e in term 1.
I20260928 13:08:56.630098 16251 raft_consensus.cc:2454] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 811f434ce8b9435c924097885cbe163e in term 1.
I20260928 13:08:56.630313 16472 leader_election.cc:302] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3153063587854129a678e89dd6c06754, 811f434ce8b9435c924097885cbe163e; no voters:
I20260928 13:08:56.630452 16619 raft_consensus.cc:2781] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:56.630587 16619 raft_consensus.cc:686] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [term 1 LEADER]: Becoming Leader. State: Replica: 811f434ce8b9435c924097885cbe163e, State: Running, Role: LEADER
I20260928 13:08:56.630645 16619 consensus_queue.cc:237] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } }
I20260928 13:08:56.631348 16101 catalog_manager.cc:5495] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e reported cstate change: term changed from 0 to 1, leader changed from <none> to 811f434ce8b9435c924097885cbe163e (127.8.88.131). New cstate: current_term: 1 leader_uuid: "811f434ce8b9435c924097885cbe163e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "811f434ce8b9435c924097885cbe163e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 39755 } health_report { overall_health: HEALTHY } } }
I20260928 13:08:56.652333 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:56.647174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.8.88.131:39755)
0928 13:08:56.649826 (+ 2652us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:56.649828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:56.649837 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:08:56.649861 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:56.649862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:56.649871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:56.649979 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:56.650124 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.650689 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.651525 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.651753 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.651835 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:56.651997 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:56.652033 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:56.652102 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:56.652186 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 13:08:56.652220 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2633,"thread_start_us":76,"threads_started":1}
I20260928 13:08:56.657577 16611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:56.655085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:38001 (local address 127.8.88.129:39089)
0928 13:08:56.655145 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:56.655147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:56.655173 (+ 26us) server_negotiation.cc:407] Connection header received
0928 13:08:56.655250 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:56.655251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:56.655262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:56.655307 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:56.655413 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.655888 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.656673 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.656949 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.657041 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:56.657228 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:56.657263 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:56.657329 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:56.657385 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 13:08:56.657490 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 13:08:56.657583 16693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:56.654968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:39089 (local address 127.8.88.131:38001)
0928 13:08:56.655111 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:56.655123 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:56.655187 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:56.655307 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:56.655307 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:56.655309 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 13:08:56.655356 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:56.655357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.655980 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.655981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:56.656574 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:56.656576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.657043 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.657044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:56.657115 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:56.657130 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 13:08:56.657329 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 13:08:56.657332 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:08:56.657381 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 13:08:56.657507 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":83,"threads_started":1}
I20260928 13:08:56.659850 16398 raft_consensus.cc:1261] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Refusing update from remote peer 811f434ce8b9435c924097885cbe163e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:56.659879 16251 raft_consensus.cc:1261] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Refusing update from remote peer 811f434ce8b9435c924097885cbe163e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:08:56.660090 16619 consensus_queue.cc:1035] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3153063587854129a678e89dd6c06754" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:56.660262 16688 consensus_queue.cc:1035] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9127714d0c1940198ed545e9565ef40f" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:08:56.667500 16690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:56.664996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:36301 (local address 127.8.88.131:39755)
0928 13:08:56.665036 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:56.665038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:56.665065 (+ 27us) server_negotiation.cc:407] Connection header received
0928 13:08:56.665133 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:56.665135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:56.665146 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:56.665207 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:08:56.665351 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.665849 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.666591 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.666879 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.666958 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:56.667161 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:08:56.667199 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:08:56.667254 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:56.667377 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 13:08:56.667412 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:08:56.667500 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:56.664957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:39755 (local address 127.8.88.129:36301)
0928 13:08:56.665019 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:56.665032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:56.665080 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:56.665216 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:56.665221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:56.665223 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 13:08:56.665292 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:56.665294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.665937 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.665938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:56.666501 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:56.666503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:56.666955 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:56.666956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:56.667017 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:56.667025 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 13:08:56.667302 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 13:08:56.667303 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 13:08:56.667349 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 13:08:56.667407 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 13:08:56.673933 16100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:56.674042 16100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:08:56.674691 16100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e6d32b09b1d49188fbb4d026acaf982: Sending DeleteTablet for 3 replicas of tablet 40dba7cbeb5a4da0af3d08a38b099c0e
I20260928 13:08:56.674957 16234 tablet_service.cc:1514] Processing DeleteTablet for tablet 40dba7cbeb5a4da0af3d08a38b099c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:56 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:49484
I20260928 13:08:56.674957 16378 tablet_service.cc:1514] Processing DeleteTablet for tablet 40dba7cbeb5a4da0af3d08a38b099c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:56 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:40224
I20260928 13:08:56.675065 16634 tablet_replica.cc:330] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754: stopping tablet replica
I20260928 13:08:56.675125 16634 raft_consensus.cc:2229] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:56.675168 16634 raft_consensus.cc:2258] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:56.675220 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 16173
I20260928 13:08:56.675258 16635 tablet_replica.cc:330] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f: stopping tablet replica
I20260928 13:08:56.675302 16635 raft_consensus.cc:2229] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:08:56.675344 16635 raft_consensus.cc:2258] T 40dba7cbeb5a4da0af3d08a38b099c0e P 9127714d0c1940198ed545e9565ef40f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:56.677073 16518 tablet_service.cc:1514] Processing DeleteTablet for tablet 40dba7cbeb5a4da0af3d08a38b099c0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:08:56 UTC) from {username='oryx', principal='oryx/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:50458
I20260928 13:08:56.677186 16636 tablet_replica.cc:330] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e: stopping tablet replica
I20260928 13:08:56.677258 16636 raft_consensus.cc:2229] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:08:56.677340 16636 raft_consensus.cc:2258] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:08:56.677543 16634 ts_tablet_manager.cc:1905] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:56.677635 16636 ts_tablet_manager.cc:1905] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:08:56.679265 16636 ts_tablet_manager.cc:1918] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:56.679284 16634 ts_tablet_manager.cc:1918] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 13:08:56.679333 16636 log.cc:1196] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/40dba7cbeb5a4da0af3d08a38b099c0e
I20260928 13:08:56.679373 16634 log.cc:1196] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/40dba7cbeb5a4da0af3d08a38b099c0e
I20260928 13:08:56.679551 16636 ts_tablet_manager.cc:1939] T 40dba7cbeb5a4da0af3d08a38b099c0e P 811f434ce8b9435c924097885cbe163e: Deleting consensus metadata
I20260928 13:08:56.679620 16634 ts_tablet_manager.cc:1939] T 40dba7cbeb5a4da0af3d08a38b099c0e P 3153063587854129a678e89dd6c06754: Deleting consensus metadata
I20260928 13:08:56.680048 16087 catalog_manager.cc:4841] TS 811f434ce8b9435c924097885cbe163e (127.8.88.131:39755): tablet 40dba7cbeb5a4da0af3d08a38b099c0e (table test-table [id=641e8c5db216485fa876285e91a324de]) successfully deleted
I20260928 13:08:56.680190 16088 catalog_manager.cc:4841] TS 3153063587854129a678e89dd6c06754 (127.8.88.130:42967): tablet 40dba7cbeb5a4da0af3d08a38b099c0e (table test-table [id=641e8c5db216485fa876285e91a324de]) successfully deleted
W20260928 13:08:56.684046 16088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:39089 (error 108)
I20260928 13:08:56.684273 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 16313
I20260928 13:08:56.689131 16698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:56.684954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:39089 (local address 127.0.0.1:49520)
0928 13:08:56.689018 (+ 4064us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:56.689081 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4039,"thread_start_us":43,"threads_started":1}
W20260928 13:08:56.689332 16088 catalog_manager.cc:4584] TS 9127714d0c1940198ed545e9565ef40f (127.8.88.129:39089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40dba7cbeb5a4da0af3d08a38b099c0e: Network error: Client connection negotiation failed: client connection to 127.8.88.129:39089: connect: Connection refused (error 111)
I20260928 13:08:56.690282 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 16453
I20260928 13:08:56.696027 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 16069
2026-09-28T13:08:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2944 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:08:56 dist-test-slave-p1lc krb5kdc[16701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:08:56 dist-test-slave-p1lc krb5kdc[16701](info): set up 2 sockets
Sep 28 13:08:56 dist-test-slave-p1lc krb5kdc[16701](info): commencing operation
krb5kdc: starting...
W20260928 13:08:57.801595 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:08:57 dist-test-slave-p1lc krb5kdc[16701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:08:57.936689 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:37309
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:38427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:37309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:58.020100 16714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:58.020325 16714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:58.020350 16714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:58.021672 16714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:58.021720 16714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:58.021735 16714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:58.021749 16714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:58.021760 16714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:58.023280 16714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:38427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:37309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:37309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:58.023497 16714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:58.023706 16714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:08:58.029030 16714 server_base.cc:1034] running on GCE node
W20260928 13:08:58.028999 16720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:58.029009 16719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:58.029173 16722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:58.029681 16714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:58.030148 16714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:58.031350 16714 hybrid_clock.cc:648] HybridClock initialized: now 1790600938031312 us; error 57 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T13:08:58Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1337 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:08:58 dist-test-slave-p1lc krb5kdc[16727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:08:58 dist-test-slave-p1lc krb5kdc[16727](info): set up 2 sockets
Sep 28 13:08:58 dist-test-slave-p1lc krb5kdc[16727](info): commencing operation
krb5kdc: starting...
W20260928 13:08:59.101599 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:08:59 dist-test-slave-p1lc krb5kdc[16727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:08:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:08:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:08:59.233099 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:33141
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:33141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:59.321152 16740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:59.321463 16740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:59.321506 16740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:59.322785 16740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:08:59.322872 16740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:59.322898 16740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:59.322928 16740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:08:59.322950 16740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:08:59.324581 16740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:33141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:33141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:59.324925 16740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:59.325186 16740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:59.330657 16748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:59.330649 16746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:59.330745 16740 server_base.cc:1034] running on GCE node
W20260928 13:08:59.330654 16745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:59.331132 16740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:59.331398 16740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:59.332535 16740 hybrid_clock.cc:648] HybridClock initialized: now 1790600939332514 us; error 39 us; skew 500 ppm
Sep 28 13:08:59 dist-test-slave-p1lc krb5kdc[16727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600939, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:59.335354 16740 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:08:59.335830 16740 webserver.cc:460] Webserver started at http://127.8.88.190:43683/ using document root <none> and password file <none>
I20260928 13:08:59.336087 16740 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:59.336138 16740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:59.336282 16740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:59.337500 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "b01e721a56224e44965f53d3e36bf014"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "05d9b1f9e81382e74b1ff4c842373a83"
server_key_iv: "5a7b24dc6f7b37c31b03e3b4e4fe4798"
server_key_version: "encryptionkey@0"
I20260928 13:08:59.337875 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "b01e721a56224e44965f53d3e36bf014"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "05d9b1f9e81382e74b1ff4c842373a83"
server_key_iv: "5a7b24dc6f7b37c31b03e3b4e4fe4798"
server_key_version: "encryptionkey@0"
I20260928 13:08:59.339342 16740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 13:08:59.340268 16755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:59.340454 16740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:59.340552 16740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "b01e721a56224e44965f53d3e36bf014"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "05d9b1f9e81382e74b1ff4c842373a83"
server_key_iv: "5a7b24dc6f7b37c31b03e3b4e4fe4798"
server_key_version: "encryptionkey@0"
I20260928 13:08:59.340637 16740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:59.359009 16740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:59.361496 16740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:59.361634 16740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:59.365998 16740 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:33141
I20260928 13:08:59.366325 16807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:33141 every 8 connection(s)
I20260928 13:08:59.366459 16740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:08:59.367287 16808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:08:59.368554 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 16740
I20260928 13:08:59.368645 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:08:59.368854 8546 external_mini_cluster.cc:1428] Setting key 2ff39bd3c239a8cd6135dee2681d10a9
I20260928 13:08:59.370028 16808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014: Bootstrap starting.
I20260928 13:08:59.370643 16808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014: Neither blocks nor log segments found. Creating new log.
I20260928 13:08:59.370949 16808 log.cc:824] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014: Log is configured to *not* fsync() on all Append() calls
I20260928 13:08:59.371758 16808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014: No bootstrap required, opened a new log
I20260928 13:08:59.374195 16808 raft_consensus.cc:348] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01e721a56224e44965f53d3e36bf014" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33141 } }
I20260928 13:08:59.374396 16808 raft_consensus.cc:374] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:08:59.374439 16808 raft_consensus.cc:729] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e721a56224e44965f53d3e36bf014, State: Initialized, Role: FOLLOWER
I20260928 13:08:59.374622 16808 consensus_queue.cc:260] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01e721a56224e44965f53d3e36bf014" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33141 } }
I20260928 13:08:59.374714 16808 raft_consensus.cc:388] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:08:59.374739 16808 raft_consensus.cc:482] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:08:59.374810 16808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:08:59.375403 16808 raft_consensus.cc:504] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01e721a56224e44965f53d3e36bf014" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33141 } }
I20260928 13:08:59.375536 16808 leader_election.cc:302] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b01e721a56224e44965f53d3e36bf014; no voters:
I20260928 13:08:59.375749 16808 leader_election.cc:288] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:08:59.375783 16813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:08:59.375906 16813 raft_consensus.cc:686] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 1 LEADER]: Becoming Leader. State: Replica: b01e721a56224e44965f53d3e36bf014, State: Running, Role: LEADER
I20260928 13:08:59.375991 16813 consensus_queue.cc:237] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01e721a56224e44965f53d3e36bf014" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33141 } }
I20260928 13:08:59.376050 16808 sys_catalog.cc:564] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:08:59.376327 16813 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b01e721a56224e44965f53d3e36bf014. Latest consensus state: current_term: 1 leader_uuid: "b01e721a56224e44965f53d3e36bf014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01e721a56224e44965f53d3e36bf014" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33141 } } }
I20260928 13:08:59.376410 16813 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [sys.catalog]: This master's current role is: LEADER
I20260928 13:08:59.376341 16815 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b01e721a56224e44965f53d3e36bf014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01e721a56224e44965f53d3e36bf014" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33141 } } }
I20260928 13:08:59.376590 16815 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [sys.catalog]: This master's current role is: LEADER
Sep 28 13:08:59 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
W20260928 13:08:59.378355 16828 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:08:59.378472 16828 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:08:59.378541 16818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:08:59.378742 16818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:08:59.380527 16818 catalog_manager.cc:1347] Generated new cluster ID: 7ec522d01af64d9f8f96ba126b215a3b
I20260928 13:08:59.380582 16818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:08:59.381726 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:59.369511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.8.88.190:33141)
0928 13:08:59.369712 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:59.369721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:59.369746 (+ 25us) server_negotiation.cc:407] Connection header received
0928 13:08:59.370188 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:59.370197 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:59.370256 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:59.370408 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:59.371167 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.372564 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.373493 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.374211 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.374328 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:59.377381 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:59.377386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:59.377386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:59.377399 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:59.379709 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:59.380197 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:59.380200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:59.380201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:59.380314 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:59.380563 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:59.380564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:59.380565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:59.380904 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:59.381052 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:59.381315 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 13:08:59.381483 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260928 13:08:59.394366 16818 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:08:59.395076 16818 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:08:59.400133 16818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014: Generated new TSK 0
I20260928 13:08:59.400569 16818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:08:59.432633 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:33141
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:59.520941 16834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:59.521188 16834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:59.521224 16834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:59.522451 16834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:59.522497 16834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:59.522526 16834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:08:59.524022 16834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:59.524248 16834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:59.524446 16834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:59.530150 16839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:59.530217 16840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:59.530158 16842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:59.530488 16834 server_base.cc:1034] running on GCE node
I20260928 13:08:59.530710 16834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:59.531016 16834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:59.532191 16834 hybrid_clock.cc:648] HybridClock initialized: now 1790600939532175 us; error 30 us; skew 500 ppm
Sep 28 13:08:59 dist-test-slave-p1lc krb5kdc[16727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600939, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:59.535210 16834 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:08:59.535744 16834 webserver.cc:460] Webserver started at http://127.8.88.129:44401/ using document root <none> and password file <none>
I20260928 13:08:59.536015 16834 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:59.536080 16834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:59.536216 16834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:59.537792 16834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "6a61f2574b79494da6eb3200f47d4efd"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "6e32ff9ffc0bb0a62045fe18732eecc5"
server_key_iv: "39eab14379ba5f02e08ba48393bf58bf"
server_key_version: "encryptionkey@0"
I20260928 13:08:59.538204 16834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "6a61f2574b79494da6eb3200f47d4efd"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "6e32ff9ffc0bb0a62045fe18732eecc5"
server_key_iv: "39eab14379ba5f02e08ba48393bf58bf"
server_key_version: "encryptionkey@0"
I20260928 13:08:59.539800 16834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 13:08:59.540844 16849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:59.541225 16834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:59.541308 16834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "6a61f2574b79494da6eb3200f47d4efd"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "6e32ff9ffc0bb0a62045fe18732eecc5"
server_key_iv: "39eab14379ba5f02e08ba48393bf58bf"
server_key_version: "encryptionkey@0"
I20260928 13:08:59.541414 16834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:59.552568 16834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:59.555194 16834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:59.555346 16834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:59.555619 16834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:59.555962 16834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:59.556028 16834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:59.556072 16834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:59.556100 16834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:59.562744 16834 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:40117
I20260928 13:08:59.562798 16962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:40117 every 8 connection(s)
I20260928 13:08:59.563172 16834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:08:59.568209 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 16834
I20260928 13:08:59.568329 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:08:59.568557 8546 external_mini_cluster.cc:1428] Setting key 4418d5b5d6219a8c0a6fd4325904c6ef
Sep 28 13:08:59 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600939, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:59.575115 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:59.564073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:60997 (local address 127.8.88.190:33141)
0928 13:08:59.564163 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:59.564170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:59.564394 (+ 224us) server_negotiation.cc:407] Connection header received
0928 13:08:59.565185 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:59.565186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:59.565198 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:59.565252 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:59.566260 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.566829 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.568011 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.568357 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.568533 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:59.571643 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:59.571645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:59.571645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:59.571653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:59.573436 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:59.573961 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:59.573963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:59.573963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:59.574031 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:59.574330 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:59.574331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:59.574331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:59.574481 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:59.574559 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:59.574903 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 13:08:59.575019 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 13:08:59.575343 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:59.564044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:60997)
0928 13:08:59.564217 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:59.564262 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:59.564823 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:59.565326 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:59.565329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:59.565935 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:59.566148 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:59.566154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.566943 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.566944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:59.567849 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:59.567852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.568471 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.568472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:59.568633 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:59.569536 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:59.569575 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:59.571525 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:59.573557 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:59.573569 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:59.573572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:59.573863 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:59.574112 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:59.574113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:59.574113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:59.574251 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:59.574564 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:59.574570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:59.574745 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 13:08:59.574871 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 13:08:59.575062 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260928 13:08:59.575930 16963 heartbeater.cc:347] Connected to a master server at 127.8.88.190:33141
I20260928 13:08:59.576076 16963 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:59.576300 16963 heartbeater.cc:511] Master 127.8.88.190:33141 requested a full tablet report, sending...
I20260928 13:08:59.576937 16772 ts_manager.cc:194] Registered new tserver with Master: 6a61f2574b79494da6eb3200f47d4efd (127.8.88.129:40117)
I20260928 13:08:59.577662 16772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:60997
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:08:59.603289 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:33141
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:59.692168 16968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:59.692415 16968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:59.692443 16968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:59.694293 16968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:59.694401 16968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:59.694442 16968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:08:59.697844 16968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:59.698230 16968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:59.698648 16968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:59.704341 16974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:59.704362 16973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:59.704347 16976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:59.704630 16968 server_base.cc:1034] running on GCE node
I20260928 13:08:59.704941 16968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:59.705173 16968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:59.706357 16968 hybrid_clock.cc:648] HybridClock initialized: now 1790600939706337 us; error 33 us; skew 500 ppm
Sep 28 13:08:59 dist-test-slave-p1lc krb5kdc[16727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600939, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:59.709226 16968 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:08:59.709704 16968 webserver.cc:460] Webserver started at http://127.8.88.130:43351/ using document root <none> and password file <none>
I20260928 13:08:59.709960 16968 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:59.710032 16968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:59.710188 16968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:59.711356 16968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "33136f9eb7f24977899ccf396029e9f6"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "4578abd8985ec37bd9bab65b9e2e7a74"
server_key_iv: "dc1b8814459ab4caaab3f5965a967686"
server_key_version: "encryptionkey@0"
I20260928 13:08:59.711907 16968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "33136f9eb7f24977899ccf396029e9f6"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "4578abd8985ec37bd9bab65b9e2e7a74"
server_key_iv: "dc1b8814459ab4caaab3f5965a967686"
server_key_version: "encryptionkey@0"
I20260928 13:08:59.713752 16968 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 13:08:59.714936 16983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:59.715171 16968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:59.715286 16968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "33136f9eb7f24977899ccf396029e9f6"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "4578abd8985ec37bd9bab65b9e2e7a74"
server_key_iv: "dc1b8814459ab4caaab3f5965a967686"
server_key_version: "encryptionkey@0"
I20260928 13:08:59.715377 16968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:59.732609 16968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:59.735198 16968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:59.735386 16968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:59.735642 16968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:59.735972 16968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:59.736030 16968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:59.736074 16968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:59.736100 16968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:59.742398 16968 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:34699
I20260928 13:08:59.742499 17096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:34699 every 8 connection(s)
I20260928 13:08:59.742865 16968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:08:59.749740 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 16968
I20260928 13:08:59.749848 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:08:59.750058 8546 external_mini_cluster.cc:1428] Setting key 6f5281f2b274e951f3909c71b404505e
Sep 28 13:08:59 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600939, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:59.753918 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:59.743700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:40215 (local address 127.8.88.190:33141)
0928 13:08:59.743744 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:59.743746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:59.743963 (+ 217us) server_negotiation.cc:407] Connection header received
0928 13:08:59.744589 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:59.744590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:59.744603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:59.744681 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:59.746209 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.746756 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.747900 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.748144 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.748242 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:59.750802 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:59.750804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:59.750804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:59.750813 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:59.752327 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:59.752842 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:59.752844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:59.752844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:59.752935 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:59.753215 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:59.753216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:59.753216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:59.753341 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:59.753422 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:59.753729 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 13:08:59.753820 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:08:59.754146 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:59.743688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:40215)
0928 13:08:59.743835 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:59.743873 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:59.744453 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:59.744763 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:59.744767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:59.745759 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:59.746110 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:59.746117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.746883 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.746884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:59.747784 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:59.747786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.748253 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.748254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:59.748420 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:59.749007 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:59.749016 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:59.750713 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:59.752447 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:59.752456 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:59.752458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:59.752752 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:59.753011 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:59.753012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:59.753013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:59.753140 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:59.753413 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:59.753420 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:59.753587 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 13:08:59.753709 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 13:08:59.753886 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260928 13:08:59.754866 17097 heartbeater.cc:347] Connected to a master server at 127.8.88.190:33141
I20260928 13:08:59.754994 17097 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:59.755213 17097 heartbeater.cc:511] Master 127.8.88.190:33141 requested a full tablet report, sending...
I20260928 13:08:59.755662 16772 ts_manager.cc:194] Registered new tserver with Master: 33136f9eb7f24977899ccf396029e9f6 (127.8.88.130:34699)
I20260928 13:08:59.756086 16772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:40215
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:08:59.782708 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:33141
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:08:59.872825 17102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:08:59.873157 17102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:08:59.873184 17102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:08:59.874436 17102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:08:59.874496 17102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:08:59.874526 17102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:08:59.876539 17102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:08:59.876895 17102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:08:59.877183 17102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:08:59.883684 17107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:59.883713 17110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:08:59.883698 17108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:08:59.884020 17102 server_base.cc:1034] running on GCE node
I20260928 13:08:59.884215 17102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:08:59.884465 17102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:08:59.885766 17102 hybrid_clock.cc:648] HybridClock initialized: now 1790600939885748 us; error 31 us; skew 500 ppm
Sep 28 13:08:59 dist-test-slave-p1lc krb5kdc[16727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600939, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:08:59.889132 17102 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:08:59.889605 17102 webserver.cc:460] Webserver started at http://127.8.88.131:45707/ using document root <none> and password file <none>
I20260928 13:08:59.889883 17102 fs_manager.cc:359] Metadata directory not provided
I20260928 13:08:59.889932 17102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:08:59.890071 17102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:08:59.891260 17102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "388f8203d2844a13bcee4b70eaae7536"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "8e5ad085e6b02e715967deeabc0fc9f7"
server_key_iv: "03f90ea32b3b1f9ab30c0ea4138ee36e"
server_key_version: "encryptionkey@0"
I20260928 13:08:59.891685 17102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "388f8203d2844a13bcee4b70eaae7536"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "8e5ad085e6b02e715967deeabc0fc9f7"
server_key_iv: "03f90ea32b3b1f9ab30c0ea4138ee36e"
server_key_version: "encryptionkey@0"
I20260928 13:08:59.893225 17102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:08:59.894171 17117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:59.894361 17102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:08:59.894469 17102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "388f8203d2844a13bcee4b70eaae7536"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "8e5ad085e6b02e715967deeabc0fc9f7"
server_key_iv: "03f90ea32b3b1f9ab30c0ea4138ee36e"
server_key_version: "encryptionkey@0"
I20260928 13:08:59.894627 17102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:08:59.902626 17102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:08:59.905251 17102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:08:59.905449 17102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:08:59.905730 17102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:08:59.906095 17102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:08:59.906162 17102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:59.906207 17102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:08:59.906234 17102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:08:59.913527 17102 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:39385
I20260928 13:08:59.913602 17230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:39385 every 8 connection(s)
I20260928 13:08:59.913987 17102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:08:59.918529 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 17102
I20260928 13:08:59.918695 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:08:59.918911 8546 external_mini_cluster.cc:1428] Setting key a470faafcc9a045b734df4c09625e3dd
Sep 28 13:08:59 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600939, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:08:59.926370 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:59.914872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:46243 (local address 127.8.88.190:33141)
0928 13:08:59.914940 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 13:08:59.914942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:08:59.915316 (+ 374us) server_negotiation.cc:407] Connection header received
0928 13:08:59.915937 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:08:59.915938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:08:59.915952 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:08:59.916055 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:08:59.917377 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.917952 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.919465 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.919808 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.919888 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:59.922850 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:08:59.922852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:08:59.922852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:08:59.922860 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:08:59.924525 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:59.925206 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:59.925208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:59.925208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:59.925277 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:08:59.925627 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:08:59.925628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:08:59.925629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:08:59.925774 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:08:59.925837 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 13:08:59.926219 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 13:08:59.926296 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 13:08:59.926719 17233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:08:59.914933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:46243)
0928 13:08:59.915160 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 13:08:59.915201 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 13:08:59.915805 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:08:59.916139 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:08:59.916148 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:08:59.916783 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:08:59.917229 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:59.917236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.918072 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.918074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:59.919292 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:08:59.919297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:08:59.919910 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:08:59.919912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:08:59.920121 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:08:59.920698 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:08:59.920707 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:08:59.922676 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:08:59.924632 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:59.924641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:59.924643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:59.925096 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:59.925377 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:08:59.925378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:08:59.925379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:08:59.925520 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:08:59.925863 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:08:59.925870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:08:59.926037 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 13:08:59.926130 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 13:08:59.926352 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
I20260928 13:08:59.927419 17231 heartbeater.cc:347] Connected to a master server at 127.8.88.190:33141
I20260928 13:08:59.927546 17231 heartbeater.cc:464] Registering TS with master...
I20260928 13:08:59.927788 17231 heartbeater.cc:511] Master 127.8.88.190:33141 requested a full tablet report, sending...
I20260928 13:08:59.928275 16772 ts_manager.cc:194] Registered new tserver with Master: 388f8203d2844a13bcee4b70eaae7536 (127.8.88.131:39385)
I20260928 13:08:59.928679 16772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:46243
I20260928 13:08:59.931066 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab.
Entry for principal oryx/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab.
WARNING: no policy specified for oryx/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab.
Entry for principal oryx/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab.
WARNING: no policy specified for oryx/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab.
Entry for principal oryx/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab.
I20260928 13:09:00.038226 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 16834
I20260928 13:09:00.043195 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 16968
I20260928 13:09:00.048540 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 17102
I20260928 13:09:00.053498 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 16740
I20260928 13:09:00.059540 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:33141
--webserver_interface=127.8.88.190
--webserver_port=43683
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:33141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:09:00.147312 17240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:00.147569 17240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:00.147595 17240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:00.148855 17240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:09:00.148944 17240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:00.148960 17240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:00.148974 17240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:09:00.148988 17240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:09:00.150573 17240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:33141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:33141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=43683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:00.150807 17240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:00.150997 17240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:00.157595 17245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:00.157593 17248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:00.157742 17246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:00.157989 17240 server_base.cc:1034] running on GCE node
I20260928 13:09:00.158262 17240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:00.158605 17240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:00.159788 17240 hybrid_clock.cc:648] HybridClock initialized: now 1790600940159757 us; error 58 us; skew 500 ppm
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600940, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:09:00.162612 17240 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:09:00.163151 17240 webserver.cc:460] Webserver started at http://127.8.88.190:43683/ using document root <none> and password file <none>
I20260928 13:09:00.163419 17240 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:00.163512 17240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:00.165114 17240 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:09:00.165874 17255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:00.166163 17240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:09:00.166280 17240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "b01e721a56224e44965f53d3e36bf014"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "05d9b1f9e81382e74b1ff4c842373a83"
server_key_iv: "5a7b24dc6f7b37c31b03e3b4e4fe4798"
server_key_version: "encryptionkey@0"
I20260928 13:09:00.166697 17240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:00.178588 17240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:00.181442 17240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:00.181640 17240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:00.186163 17240 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:33141
I20260928 13:09:00.186245 17307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:33141 every 8 connection(s)
I20260928 13:09:00.186673 17240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:09:00.188283 17308 sys_catalog.cc:263] Verifying existing consensus state
I20260928 13:09:00.189064 17308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014: Bootstrap starting.
I20260928 13:09:00.195523 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 17240
I20260928 13:09:00.195871 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:40117
--local_ip_for_outbound_sockets=127.8.88.129
--tserver_master_addrs=127.8.88.190:33141
--webserver_port=44401
--webserver_interface=127.8.88.129
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.88.129
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:09:00.197633 17308 log.cc:824] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014: Log is configured to *not* fsync() on all Append() calls
I20260928 13:09:00.199393 17308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:09:00.199697 17308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014: Bootstrap complete.
I20260928 13:09:00.202134 17308 raft_consensus.cc:348] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01e721a56224e44965f53d3e36bf014" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33141 } }
I20260928 13:09:00.202400 17308 raft_consensus.cc:729] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01e721a56224e44965f53d3e36bf014, State: Initialized, Role: FOLLOWER
I20260928 13:09:00.202627 17308 consensus_queue.cc:260] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01e721a56224e44965f53d3e36bf014" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33141 } }
I20260928 13:09:00.202713 17308 raft_consensus.cc:388] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:09:00.202767 17308 raft_consensus.cc:482] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:09:00.202816 17308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 1 FOLLOWER]: Advancing to term 2
I20260928 13:09:00.203897 17308 raft_consensus.cc:504] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01e721a56224e44965f53d3e36bf014" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33141 } }
I20260928 13:09:00.204033 17308 leader_election.cc:302] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b01e721a56224e44965f53d3e36bf014; no voters:
I20260928 13:09:00.204257 17308 leader_election.cc:288] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 13:09:00.204295 17312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 13:09:00.204409 17312 raft_consensus.cc:686] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [term 2 LEADER]: Becoming Leader. State: Replica: b01e721a56224e44965f53d3e36bf014, State: Running, Role: LEADER
I20260928 13:09:00.204491 17312 consensus_queue.cc:237] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01e721a56224e44965f53d3e36bf014" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33141 } }
I20260928 13:09:00.204559 17308 sys_catalog.cc:564] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:09:00.204747 17312 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b01e721a56224e44965f53d3e36bf014. Latest consensus state: current_term: 2 leader_uuid: "b01e721a56224e44965f53d3e36bf014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01e721a56224e44965f53d3e36bf014" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33141 } } }
I20260928 13:09:00.204845 17312 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [sys.catalog]: This master's current role is: LEADER
I20260928 13:09:00.204753 17313 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b01e721a56224e44965f53d3e36bf014" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01e721a56224e44965f53d3e36bf014" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33141 } } }
I20260928 13:09:00.205087 17313 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014 [sys.catalog]: This master's current role is: LEADER
I20260928 13:09:00.205224 17316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:09:00.205772 17316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:09:00.205965 17316 catalog_manager.cc:1259] Loaded cluster ID: 7ec522d01af64d9f8f96ba126b215a3b
I20260928 13:09:00.206156 17316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:09:00.206969 17316 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:09:00.207182 17316 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b01e721a56224e44965f53d3e36bf014: Loaded TSK: 0
I20260928 13:09:00.207410 17316 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 13:09:00.297586 17310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:00.297834 17310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:00.297861 17310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:00.299265 17310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:00.299324 17310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:00.299355 17310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:09:00.301033 17310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.129.keytab
--principal=oryx/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:40117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=44401
--tserver_master_addrs=127.8.88.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:00.301311 17310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:00.301578 17310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:00.307636 17334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:00.307636 17335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:00.307859 17310 server_base.cc:1034] running on GCE node
W20260928 13:09:00.307723 17337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:00.308199 17310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:00.308522 17310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:00.309718 17310 hybrid_clock.cc:648] HybridClock initialized: now 1790600940309705 us; error 31 us; skew 500 ppm
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600940, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:09:00.313004 17310 init.cc:377] Logged in from keytab as oryx/127.8.88.129@KRBTEST.COM (short username oryx)
I20260928 13:09:00.313529 17310 webserver.cc:460] Webserver started at http://127.8.88.129:44401/ using document root <none> and password file <none>
I20260928 13:09:00.313804 17310 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:00.313951 17310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:00.315500 17310 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:09:00.316509 17344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:00.316728 17310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:09:00.316841 17310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "6a61f2574b79494da6eb3200f47d4efd"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "6e32ff9ffc0bb0a62045fe18732eecc5"
server_key_iv: "39eab14379ba5f02e08ba48393bf58bf"
server_key_version: "encryptionkey@0"
I20260928 13:09:00.317315 17310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:00.334025 17310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:00.336513 17310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:00.336668 17310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:00.337019 17310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:09:00.337440 17310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:09:00.337484 17310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:00.337508 17310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:09:00.337522 17310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:00.344303 17310 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:40117
I20260928 13:09:00.344349 17457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:40117 every 8 connection(s)
I20260928 13:09:00.344738 17310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:09:00.351760 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 17310
I20260928 13:09:00.352136 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:34699
--local_ip_for_outbound_sockets=127.8.88.130
--tserver_master_addrs=127.8.88.190:33141
--webserver_port=43351
--webserver_interface=127.8.88.130
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.88.130
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:00.356108 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:00.345525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:49393)
0928 13:09:00.345745 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:00.345799 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:00.346442 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:00.347126 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:00.347130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:00.347729 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:00.347950 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.347957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.349714 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.349715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.350712 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.350715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.351682 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.351684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.351841 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.352569 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:00.352580 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:00.355880 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
W20260928 13:09:00.356370 17458 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:33141 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.88.190:33141: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
W20260928 13:09:00.356741 17461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:00.345668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:49393 (local address 127.8.88.190:33141)
0928 13:09:00.346652 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:00.346661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:00.346701 (+ 40us) server_negotiation.cc:407] Connection header received
0928 13:09:00.346838 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:00.346844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:00.346902 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:00.347054 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:00.348092 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.349566 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.350825 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.351466 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.351592 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.356085 (+ 4493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:49393: BlockingRecv error: recv got EOF from 127.8.88.129:49393 (error 108)
Metrics: {"server-negotiator.queue_time_us":929,"thread_start_us":94,"threads_started":1}
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:00.363744 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:00.356848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:48063)
0928 13:09:00.357268 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:00.357280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:00.357347 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:00.357793 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:00.357794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:00.358112 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:00.358198 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.358202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.359313 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.359314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.360263 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.360266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.360726 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.360727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.360817 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.361383 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:00.361391 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:00.363642 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":342,"threads_started":1}
W20260928 13:09:00.363804 17461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:00.356977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:48063 (local address 127.8.88.190:33141)
0928 13:09:00.357049 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:00.357052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:00.357331 (+ 279us) server_negotiation.cc:407] Connection header received
0928 13:09:00.357410 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:00.357412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:00.357432 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:00.357710 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:00.358334 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.359207 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.360365 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.360630 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.360703 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.363732 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:48063: BlockingRecv error: recv got EOF from 127.8.88.129:48063 (error 108)
Metrics: {"server-negotiator.queue_time_us":53,"thread_start_us":92,"threads_started":1}
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:00.368672 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:00.364019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:38971)
0928 13:09:00.364069 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:00.364079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:00.364116 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:00.364286 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:00.364288 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:00.364431 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:00.364467 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.364468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.365315 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.365317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.365832 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.365833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.366230 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.366231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.366294 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.366823 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:00.366828 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:00.368595 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 13:09:00.368817 17461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:00.364102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:38971 (local address 127.8.88.190:33141)
0928 13:09:00.364155 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:00.364158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:00.364166 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:09:00.364189 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:00.364190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:00.364201 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:00.364279 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:00.364557 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.365197 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.365957 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.366145 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.366229 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.368749 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:38971: BlockingRecv error: recv got EOF from 127.8.88.129:38971 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 13:09:00.368956 17458 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 13:09:00.444624 17462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:00.444967 17462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:00.445024 17462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:00.446332 17462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:00.446434 17462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:00.446483 17462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:09:00.448094 17462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.130.keytab
--principal=oryx/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:34699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=43351
--tserver_master_addrs=127.8.88.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:00.448408 17462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:00.448673 17462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:00.454679 17469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:00.454680 17470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:00.454679 17472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:00.454728 17462 server_base.cc:1034] running on GCE node
I20260928 13:09:00.455219 17462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:00.455510 17462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:00.456715 17462 hybrid_clock.cc:648] HybridClock initialized: now 1790600940456703 us; error 32 us; skew 500 ppm
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600940, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:09:00.459494 17462 init.cc:377] Logged in from keytab as oryx/127.8.88.130@KRBTEST.COM (short username oryx)
I20260928 13:09:00.460098 17462 webserver.cc:460] Webserver started at http://127.8.88.130:43351/ using document root <none> and password file <none>
I20260928 13:09:00.460356 17462 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:00.460451 17462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:00.461966 17462 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:09:00.462646 17479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:00.462823 17462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:09:00.462936 17462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "33136f9eb7f24977899ccf396029e9f6"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "4578abd8985ec37bd9bab65b9e2e7a74"
server_key_iv: "dc1b8814459ab4caaab3f5965a967686"
server_key_version: "encryptionkey@0"
I20260928 13:09:00.463308 17462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:00.494867 17462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:00.497519 17462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:00.497685 17462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:00.498000 17462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:09:00.498406 17462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:09:00.498447 17462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:00.498471 17462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:09:00.498484 17462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:00.504832 17462 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:34699
I20260928 13:09:00.504904 17592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:34699 every 8 connection(s)
I20260928 13:09:00.505363 17462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:09:00.509271 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 17462
I20260928 13:09:00.509634 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:39385
--local_ip_for_outbound_sockets=127.8.88.131
--tserver_master_addrs=127.8.88.190:33141
--webserver_port=45707
--webserver_interface=127.8.88.131
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.88.131
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:00.515067 17595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:00.506133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:39279)
0928 13:09:00.506361 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:00.506408 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:00.506919 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:00.507136 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:00.507139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:00.507711 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:00.507934 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.507938 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.508604 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.508605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.509636 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.509638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.510168 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.510170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.510283 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.511211 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:00.511224 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:00.514849 (+ 3625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":106,"threads_started":1}
W20260928 13:09:00.515125 17461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:00.506166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:39279 (local address 127.8.88.190:33141)
0928 13:09:00.506258 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:00.506261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:00.506503 (+ 242us) server_negotiation.cc:407] Connection header received
0928 13:09:00.507013 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:00.507013 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:00.507029 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:00.507082 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:00.508032 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.508517 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.509732 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.510058 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.510139 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.515028 (+ 4889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:39279: BlockingRecv error: recv got EOF from 127.8.88.130:39279 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 13:09:00.515558 17593 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:33141 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.88.190:33141: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:00.521487 17595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:00.515911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:42783)
0928 13:09:00.515950 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:00.515961 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:00.516010 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:00.516166 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:00.516167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:00.516324 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:00.516390 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.516393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.517157 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.517158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.517695 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.517697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.518095 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.518095 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.518160 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.518741 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:00.518749 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:00.521415 (+ 2666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":17}
W20260928 13:09:00.521603 17461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:00.515965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:42783 (local address 127.8.88.190:33141)
0928 13:09:00.516044 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:00.516050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:00.516059 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:09:00.516087 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:00.516088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:00.516097 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:00.516150 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:00.516479 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.517064 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.517802 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.518015 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.518097 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.521538 (+ 3441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:42783: BlockingRecv error: recv got EOF from 127.8.88.130:42783 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:00.527027 17595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:00.521836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:33515)
0928 13:09:00.521951 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:00.521964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:00.522019 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:00.522159 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:00.522159 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:00.522317 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:00.522362 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.522364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.523012 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.523014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.523890 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.523893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.524299 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.524300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.524359 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.524821 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:00.524827 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:00.526951 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260928 13:09:00.527151 17461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:00.521904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:33515 (local address 127.8.88.190:33141)
0928 13:09:00.521947 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:00.521950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:00.522002 (+ 52us) server_negotiation.cc:407] Connection header received
0928 13:09:00.522086 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:00.522087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:00.522095 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:00.522136 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:00.522434 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.522901 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.524009 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.524221 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.524285 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.527058 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:33515: BlockingRecv error: recv got EOF from 127.8.88.130:33515 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 13:09:00.527205 17593 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 13:09:00.598923 17596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:00.599179 17596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:00.599205 17596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:00.600673 17596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:00.600746 17596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:00.600782 17596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:09:00.602526 17596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/krb5kdc/oryx_127.8.88.131.keytab
--principal=oryx/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:39385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=45707
--tserver_master_addrs=127.8.88.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:00.602766 17596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:00.602973 17596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:00.608732 17601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:00.608732 17604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:00.608815 17602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:00.608990 17596 server_base.cc:1034] running on GCE node
I20260928 13:09:00.609258 17596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:00.609553 17596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:00.610731 17596 hybrid_clock.cc:648] HybridClock initialized: now 1790600940610721 us; error 31 us; skew 500 ppm
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600940, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:09:00.613718 17596 init.cc:377] Logged in from keytab as oryx/127.8.88.131@KRBTEST.COM (short username oryx)
I20260928 13:09:00.614280 17596 webserver.cc:460] Webserver started at http://127.8.88.131:45707/ using document root <none> and password file <none>
I20260928 13:09:00.614544 17596 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:00.614637 17596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:00.616283 17596 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:09:00.617134 17611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:00.617631 17596 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:09:00.617785 17596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "388f8203d2844a13bcee4b70eaae7536"
format_stamp: "Formatted at 2026-09-28 13:08:59 on dist-test-slave-p1lc"
server_key: "8e5ad085e6b02e715967deeabc0fc9f7"
server_key_iv: "03f90ea32b3b1f9ab30c0ea4138ee36e"
server_key_version: "encryptionkey@0"
I20260928 13:09:00.618273 17596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:00.636181 17596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:00.638559 17596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:00.638742 17596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:00.639006 17596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:09:00.639339 17596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:09:00.639406 17596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:00.639446 17596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:09:00.639539 17596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:00.645889 17596 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:39385
I20260928 13:09:00.645936 17724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:39385 every 8 connection(s)
I20260928 13:09:00.646358 17596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:00.654551 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:00.647138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:53679)
0928 13:09:00.647309 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:00.647356 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:00.647853 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:00.648116 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:00.648120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:00.648728 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:00.649035 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.649042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.649787 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.649787 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.650869 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.650872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.651241 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.651242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.651347 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.651928 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:00.651937 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:00.654380 (+ 2443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
W20260928 13:09:00.654556 17461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:00.647185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:53679 (local address 127.8.88.190:33141)
0928 13:09:00.647259 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:00.647260 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:00.647459 (+ 199us) server_negotiation.cc:407] Connection header received
0928 13:09:00.647977 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:00.647978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:00.647989 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:00.648056 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:00.649155 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.649670 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.650970 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.651159 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.651221 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.654506 (+ 3285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:53679: BlockingRecv error: recv got EOF from 127.8.88.131:53679 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 13:09:00.654832 17725 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:33141 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.88.190:33141: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
I20260928 13:09:00.656404 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 17596
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:00.660416 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:00.655148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:41445)
0928 13:09:00.655227 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:00.655240 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:00.655292 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:00.655628 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:00.655629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:00.655767 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:00.655861 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.655863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.656609 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.656610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.657467 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.657469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.657877 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.657878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.657955 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.658448 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:00.658454 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:00.660290 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 13:09:00.660454 17461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:00.655391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:41445 (local address 127.8.88.190:33141)
0928 13:09:00.655480 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:00.655482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:00.655491 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:09:00.655513 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:00.655513 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:00.655522 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:00.655621 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:00.655985 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.656510 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.657565 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.657765 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.657858 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.660392 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:41445: BlockingRecv error: recv got EOF from 127.8.88.131:41445 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:00.664757 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:00.656848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.8.88.190:33141)
0928 13:09:00.656952 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:00.656954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:00.657138 (+ 184us) server_negotiation.cc:407] Connection header received
0928 13:09:00.657187 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:00.657188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:00.657200 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:00.657266 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:00.657909 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.658403 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.659398 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.659625 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.659710 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.661296 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:00.661302 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:00.661302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:09:00.661316 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:00.663034 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:09:00.663456 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:09:00.663460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:09:00.663461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:09:00.663577 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:09:00.663791 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:09:00.663792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:09:00.663793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:09:00.664186 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:00.664393 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:00.664521 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 13:09:00.664645 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 13:09:00 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:00.665774 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:00.660715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:57635)
0928 13:09:00.660800 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:00.660811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:00.660861 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:00.661038 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:00.661038 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:00.661168 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:00.661197 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.661199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.662010 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.662011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.662629 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:00.662632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.663047 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.663048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:00.663106 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.663537 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:00.663542 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:00.665695 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 13:09:00.665934 17725 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 13:09:00.666033 17461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:00.660762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:57635 (local address 127.8.88.190:33141)
0928 13:09:00.660821 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:00.660823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:00.660846 (+ 23us) server_negotiation.cc:407] Connection header received
0928 13:09:00.660944 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:00.660944 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:00.660955 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:00.661035 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:00.661274 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.661902 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.662738 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:00.662927 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:00.663024 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:00.665961 (+ 2937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:57635: BlockingRecv error: recv got EOF from 127.8.88.131:57635 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 13:09:01 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:01 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:01.374601 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:01.369417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:52657)
0928 13:09:01.369561 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:01.369574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:01.369642 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:01.369793 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:01.369794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:01.369959 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:01.370010 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:01.370011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:01.370788 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:01.370789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:01.371389 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:01.371391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:01.371773 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:01.371774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:01.371815 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:01.372339 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:01.372343 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:01.374524 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
W20260928 13:09:01.374779 17730 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:01.369513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:52657 (local address 127.8.88.190:33141)
0928 13:09:01.369641 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:01.369643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:01.369654 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:09:01.369692 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:01.369693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:01.369706 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:01.369778 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:01.370080 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:01.370644 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:01.371487 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:01.371692 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:01.371768 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:01.374650 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:52657: BlockingRecv error: recv got EOF from 127.8.88.129:52657 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
Sep 28 13:09:01 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:01 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:01.533165 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:01.527747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:33583)
0928 13:09:01.527974 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:01.527988 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:01.528100 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:01.528346 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:01.528347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:01.528531 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:01.528580 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:01.528582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:01.529362 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:01.529363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:01.529924 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:01.529926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:01.530304 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:01.530305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:01.530362 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:01.530861 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:01.530866 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:01.533057 (+ 2191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
W20260928 13:09:01.533226 17730 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:01.527879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:33583 (local address 127.8.88.190:33141)
0928 13:09:01.527979 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:01.527982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:01.528079 (+ 97us) server_negotiation.cc:407] Connection header received
0928 13:09:01.528189 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:01.528190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:01.528203 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:01.528339 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:01.528680 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:01.529263 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:01.530021 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:01.530220 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:01.530305 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:01.533161 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:33583: BlockingRecv error: recv got EOF from 127.8.88.130:33583 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 13:09:01 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:01 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:01.672403 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:01.666450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:47915)
0928 13:09:01.666608 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:01.666621 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:01.666755 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:01.667005 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:01.667006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:01.667272 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:01.667323 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:01.667326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:01.668119 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:01.668120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:01.669112 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:01.669114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:01.669543 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:01.669544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:01.669600 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:01.670182 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:01.670188 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:01.672320 (+ 2132us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
W20260928 13:09:01.672629 17730 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:01.666539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:47915 (local address 127.8.88.190:33141)
0928 13:09:01.666626 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:01.666633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:01.666823 (+ 190us) server_negotiation.cc:407] Connection header received
0928 13:09:01.666865 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:01.666866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:01.666890 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:01.666942 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:01.667482 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:01.668009 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:01.669203 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:01.669457 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:01.669532 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:01.672513 (+ 2981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:47915: BlockingRecv error: recv got EOF from 127.8.88.131:47915 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 13:09:02 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:02 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:02.380415 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:02.375217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:38507)
0928 13:09:02.375406 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:02.375420 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:02.375485 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:02.375766 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:02.375768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:02.376019 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:02.376085 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:02.376088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:02.376796 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:02.376797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:02.377408 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:02.377410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:02.377786 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:02.377787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:02.377831 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:02.378329 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:02.378334 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:02.380351 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
W20260928 13:09:02.380590 17734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:02.375422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:38507 (local address 127.8.88.190:33141)
0928 13:09:02.375556 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:02.375558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:02.375568 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:09:02.375601 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:02.375602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:02.375612 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:02.375705 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:02.376184 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:02.376710 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:02.377502 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:02.377715 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:02.377773 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:02.380476 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:38507: BlockingRecv error: recv got EOF from 127.8.88.129:38507 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
Sep 28 13:09:02 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:02 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:02.538748 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:02.533700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:47319)
0928 13:09:02.533977 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:02.533991 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:02.534071 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:02.534287 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:02.534288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:02.534448 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:02.534494 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:02.534495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:02.535217 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:02.535218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:02.535746 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:02.535748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:02.536131 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:02.536132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:02.536188 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:02.536732 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:02.536737 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:02.538635 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":92,"threads_started":1}
W20260928 13:09:02.538805 17734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:02.533821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:47319 (local address 127.8.88.190:33141)
0928 13:09:02.534002 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:02.534004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:02.534067 (+ 63us) server_negotiation.cc:407] Connection header received
0928 13:09:02.534133 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:02.534134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:02.534147 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:02.534203 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:02.534582 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:02.535120 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:02.535846 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:02.536055 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:02.536134 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:02.538735 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:47319: BlockingRecv error: recv got EOF from 127.8.88.130:47319 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
Sep 28 13:09:02 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:02 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:02.677917 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:02.673013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:47975)
0928 13:09:02.673170 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:02.673187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:02.673235 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:02.673436 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:02.673437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:02.673599 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:02.673648 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:02.673650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:02.674412 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:02.674413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:02.674970 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:02.674972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:02.675372 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:02.675373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:02.675416 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:02.675954 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:02.675959 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:02.677856 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
W20260928 13:09:02.678084 17734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:02.673191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:47975 (local address 127.8.88.190:33141)
0928 13:09:02.673260 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:02.673267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:02.673276 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:09:02.673301 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:02.673302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:02.673313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:02.673377 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:02.673763 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:02.674319 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:02.675053 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:02.675299 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:02.675360 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:02.677986 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:47975: BlockingRecv error: recv got EOF from 127.8.88.131:47975 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 13:09:03 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:03 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:03.386950 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:03.381141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:51915)
0928 13:09:03.381355 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:03.381366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:03.381422 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:03.381586 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:03.381586 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:03.381818 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:03.381895 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:03.381898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:03.382837 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:03.382839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:03.383437 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:03.383439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:03.383892 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:03.383893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:03.383945 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:03.384497 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:03.384503 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:03.386873 (+ 2370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":144,"threads_started":1}
W20260928 13:09:03.387113 17738 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:03.381246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:51915 (local address 127.8.88.190:33141)
0928 13:09:03.381388 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:03.381391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:03.381412 (+ 21us) server_negotiation.cc:407] Connection header received
0928 13:09:03.381474 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:03.381474 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:03.381485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:03.381563 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:03.382019 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:03.382745 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:03.383529 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:03.383750 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:03.383810 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:03.387005 (+ 3195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:51915: BlockingRecv error: recv got EOF from 127.8.88.129:51915 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
Sep 28 13:09:03 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:03 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:03.545562 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:03.539555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:59417)
0928 13:09:03.539758 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:03.539771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:03.539878 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:03.540057 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:03.540058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:03.540311 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:03.540360 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:03.540363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:03.541416 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:03.541418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:03.542066 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:03.542068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:03.542647 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:03.542648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:03.542696 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:03.543295 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:03.543301 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:03.545441 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":118,"threads_started":1}
W20260928 13:09:03.545603 17738 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:03.539650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:59417 (local address 127.8.88.190:33141)
0928 13:09:03.539719 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:03.539722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:03.539898 (+ 176us) server_negotiation.cc:407] Connection header received
0928 13:09:03.539935 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:03.539936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:03.539952 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:03.540023 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:03.540519 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:03.541303 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:03.542362 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:03.542562 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:03.542629 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:03.545535 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:59417: BlockingRecv error: recv got EOF from 127.8.88.130:59417 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 13:09:03 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:03 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:03.683743 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:03.678495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:42329)
0928 13:09:03.678731 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:03.678744 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:03.678844 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:03.679028 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:03.679029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:03.679222 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:03.679269 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:03.679271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:03.680082 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:03.680083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:03.680670 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:03.680673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:03.681131 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:03.681132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:03.681197 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:03.681723 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:03.681728 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:03.683672 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":110,"threads_started":1}
W20260928 13:09:03.683861 17738 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:03.678629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:42329 (local address 127.8.88.190:33141)
0928 13:09:03.678721 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:03.678724 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:03.678834 (+ 110us) server_negotiation.cc:407] Connection header received
0928 13:09:03.678913 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:03.678914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:03.678929 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:03.679008 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:03.679372 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:03.679895 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:03.680771 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:03.681045 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:03.681130 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:03.683781 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:42329: BlockingRecv error: recv got EOF from 127.8.88.131:42329 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 13:09:04 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:04 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:04.398576 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:04.387622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:45981)
0928 13:09:04.387802 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:04.387813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:04.387868 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:04.388099 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:04.388100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:04.388279 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:04.388338 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:04.388339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:04.389117 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:04.389119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:04.389670 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:04.389672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:04.390236 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:04.390238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:04.390305 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:04.396241 (+ 5936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:04.396247 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:04.398514 (+ 2267us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":111,"threads_started":1}
W20260928 13:09:04.398772 17742 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:04.387750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:45981 (local address 127.8.88.190:33141)
0928 13:09:04.387900 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:04.387902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:04.387916 (+ 14us) server_negotiation.cc:407] Connection header received
0928 13:09:04.387943 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:04.387943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:04.387953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:04.388045 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:04.388458 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:04.389020 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:04.389799 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:04.390110 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:04.390178 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:04.398698 (+ 8520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:45981: BlockingRecv error: recv got EOF from 127.8.88.129:45981 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
Sep 28 13:09:04 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:04 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:04.552201 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:04.546356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:49763)
0928 13:09:04.546597 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:04.546608 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:04.546688 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:04.546922 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:04.546923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:04.547177 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:04.547233 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:04.547235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:04.548088 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:04.548090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:04.548708 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:04.548710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:04.549336 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:04.549337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:04.549388 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:04.550137 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:04.550142 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:04.552096 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
W20260928 13:09:04.552258 17742 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:04.546495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:49763 (local address 127.8.88.190:33141)
0928 13:09:04.546566 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:04.546573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:04.546700 (+ 127us) server_negotiation.cc:407] Connection header received
0928 13:09:04.546745 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:04.546746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:04.546757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:04.546831 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:04.547356 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:04.547919 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:04.548820 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:04.549120 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:04.549333 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:04.552195 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:49763: BlockingRecv error: recv got EOF from 127.8.88.130:49763 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 13:09:04 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:04 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:04.689631 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:04.684530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:45489)
0928 13:09:04.684678 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:04.684712 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:04.684777 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:04.685049 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:04.685050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:04.685233 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:04.685283 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:04.685286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:04.685990 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:04.685991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:04.686553 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:04.686555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:04.686951 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:04.686952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:04.687019 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:04.687567 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:04.687572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:04.689571 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
W20260928 13:09:04.689805 17742 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:04.684663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:45489 (local address 127.8.88.190:33141)
0928 13:09:04.684753 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:04.684755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:04.684767 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:09:04.684856 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:04.684857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:04.684901 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:04.685040 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:04.685367 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:04.685886 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:04.686643 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:04.686841 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:04.686951 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:04.689715 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:45489: BlockingRecv error: recv got EOF from 127.8.88.131:45489 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 13:09:05 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:05 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:05.404953 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:05.399195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:49575)
0928 13:09:05.399381 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:05.399404 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:05.399472 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:05.399734 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:05.399736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:05.399914 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:05.399965 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:05.399967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:05.400784 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:05.400785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:05.401468 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:05.401470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:05.401992 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:05.401994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:05.402072 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:05.402713 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:05.402718 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:05.404832 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
W20260928 13:09:05.405107 17746 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:05.399423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:49575 (local address 127.8.88.190:33141)
0928 13:09:05.399549 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:05.399551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:05.399561 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:09:05.399587 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:05.399588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:05.399602 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:05.399666 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:05.400081 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:05.400681 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:05.401577 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:05.401863 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:05.401957 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:05.404995 (+ 3038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:49575: BlockingRecv error: recv got EOF from 127.8.88.129:49575 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
Sep 28 13:09:05 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:05 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:05.558276 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:05.552863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:56355)
0928 13:09:05.553086 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:05.553101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:05.553199 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:05.553358 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:05.553359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:05.553587 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:05.553664 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:05.553666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:05.554396 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:05.554397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:05.554976 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:05.554978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:05.555379 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:05.555380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:05.555435 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:05.555963 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:05.555967 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:05.558217 (+ 2250us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":144,"threads_started":1}
W20260928 13:09:05.558465 17746 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:05.553079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:56355 (local address 127.8.88.190:33141)
0928 13:09:05.553169 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:05.553171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:05.553182 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:09:05.553287 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:05.553288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:05.553301 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:05.553353 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:05.553760 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:05.554295 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:05.555071 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:05.555288 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:05.555379 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:05.558397 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:56355: BlockingRecv error: recv got EOF from 127.8.88.130:56355 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 13:09:05 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:05 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:05.696063 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:05.690299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:44435)
0928 13:09:05.690493 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:05.690512 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:05.690587 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:05.690904 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:05.690905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:05.691222 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:05.691284 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:05.691287 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:05.692067 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:05.692068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:05.692613 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:05.692615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:05.693126 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:05.693127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:05.693181 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:05.693738 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:05.693742 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:05.695963 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":119,"threads_started":1}
W20260928 13:09:05.696126 17746 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:05.690413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:44435 (local address 127.8.88.190:33141)
0928 13:09:05.690498 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:05.690501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:05.690592 (+ 91us) server_negotiation.cc:407] Connection header received
0928 13:09:05.690691 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:05.690692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:05.690708 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:05.690780 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:05.691439 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:05.691975 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:05.692703 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:05.693004 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:05.693075 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:05.696061 (+ 2986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:44435: BlockingRecv error: recv got EOF from 127.8.88.131:44435 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 13:09:06 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:06 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:06.410578 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:06.405563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:59027)
0928 13:09:06.405699 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:06.405711 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:06.405785 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:06.405973 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:06.405974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:06.406140 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:06.406190 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:06.406192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:06.406925 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:06.406926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:06.407466 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:06.407468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:06.407880 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:06.407881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:06.407943 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:06.408551 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:06.408556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:06.410518 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
W20260928 13:09:06.410763 17750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:06.405634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:59027 (local address 127.8.88.190:33141)
0928 13:09:06.405742 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:06.405744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:06.405779 (+ 35us) server_negotiation.cc:407] Connection header received
0928 13:09:06.405828 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:06.405828 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:06.405855 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:06.405957 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:06.406258 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:06.406803 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:06.407566 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:06.407784 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:06.407845 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:06.410687 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:59027: BlockingRecv error: recv got EOF from 127.8.88.129:59027 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":61,"threads_started":1}
Sep 28 13:09:06 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:06 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:06.564029 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:06.558939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:50861)
0928 13:09:06.559104 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:06.559122 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:06.559185 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:06.559351 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:06.559352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:06.559530 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:06.559575 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:06.559577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:06.560290 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:06.560291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:06.560999 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:06.561002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:06.561439 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:06.561440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:06.561506 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:06.562097 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:06.562102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:06.563927 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
W20260928 13:09:06.564087 17750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:06.559105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:50861 (local address 127.8.88.190:33141)
0928 13:09:06.559182 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:06.559184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:06.559202 (+ 18us) server_negotiation.cc:407] Connection header received
0928 13:09:06.559243 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:06.559243 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:06.559254 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:06.559325 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:06.559652 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:06.560204 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:06.561082 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:06.561348 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:06.561427 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:06.564025 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:50861: BlockingRecv error: recv got EOF from 127.8.88.130:50861 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 13:09:06 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:06 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:06.701606 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:06.696653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:45005)
0928 13:09:06.696803 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:06.696814 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:06.696899 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:06.697029 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:06.697030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:06.697199 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:06.697244 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:06.697245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:06.697948 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:06.697949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:06.698528 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:06.698531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:06.698957 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:06.698958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:06.699006 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:06.699498 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:06.699502 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:06.701544 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
W20260928 13:09:06.701903 17750 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:06.696707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:45005 (local address 127.8.88.190:33141)
0928 13:09:06.696773 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:06.696775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:06.696886 (+ 111us) server_negotiation.cc:407] Connection header received
0928 13:09:06.696954 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:06.696955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:06.696966 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:06.697008 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:06.697322 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:06.697834 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:06.698643 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:06.698864 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:06.698928 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:06.701830 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:45005: BlockingRecv error: recv got EOF from 127.8.88.131:45005 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 13:09:07 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:07 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:07.416276 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:07.411227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:46651)
0928 13:09:07.411374 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:07.411393 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:07.411444 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:07.411734 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:07.411735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:07.411908 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:07.411961 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:07.411962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:07.412748 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:07.412749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:07.413372 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:07.413374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:07.413766 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:07.413767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:07.413821 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:07.414377 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:07.414381 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:07.416211 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
W20260928 13:09:07.416461 17754 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:07.411363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:46651 (local address 127.8.88.190:33141)
0928 13:09:07.411498 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:07.411500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:07.411514 (+ 14us) server_negotiation.cc:407] Connection header received
0928 13:09:07.411546 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:07.411547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:07.411561 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:07.411687 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:07.412097 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:07.412627 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:07.413482 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:07.413677 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:07.413732 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:07.416351 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:46651: BlockingRecv error: recv got EOF from 127.8.88.129:46651 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
Sep 28 13:09:07 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:07 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:07.570789 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:07.564773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:42337)
0928 13:09:07.565000 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:07.565014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:07.565127 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:07.565538 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:07.565539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:07.565728 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:07.565783 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:07.565785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:07.566721 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:07.566722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:07.567404 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:07.567407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:07.567889 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:07.567890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:07.567941 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:07.568506 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:07.568511 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:07.570720 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
W20260928 13:09:07.570950 17754 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:07.565057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:42337 (local address 127.8.88.190:33141)
0928 13:09:07.565363 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:07.565365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:07.565374 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:09:07.565426 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:07.565427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:07.565439 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:07.565501 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:07.565936 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:07.566611 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:07.567513 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:07.567781 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:07.567851 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:07.570853 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:42337: BlockingRecv error: recv got EOF from 127.8.88.130:42337 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 28 13:09:07 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:07 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:07.707994 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:07.702378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:48783)
0928 13:09:07.702730 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:07.702743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:07.702841 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:07.703025 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:07.703026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:07.703241 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:07.703289 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:07.703291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:07.704020 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:07.704021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:07.704624 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:07.704626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:07.705083 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:07.705084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:07.705175 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:07.705680 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:07.705686 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:07.707895 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":120,"threads_started":1}
W20260928 13:09:07.708050 17754 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:07.702518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:48783 (local address 127.8.88.190:33141)
0928 13:09:07.702726 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:07.702729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:07.702821 (+ 92us) server_negotiation.cc:407] Connection header received
0928 13:09:07.702925 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:07.702926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:07.702946 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:07.703017 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:07.703380 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:07.703936 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:07.704723 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:07.705022 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:07.705085 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:07.707984 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:48783: BlockingRecv error: recv got EOF from 127.8.88.131:48783 (error 108)
Metrics: {"server-negotiator.queue_time_us":176}
Sep 28 13:09:08 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:08 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:08.422401 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:08.417021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:35529)
0928 13:09:08.417206 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:08.417221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:08.417294 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:08.417457 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:08.417458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:08.417638 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:08.417689 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:08.417691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:08.418352 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:08.418353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:08.419012 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:08.419014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:08.419444 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:08.419446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:08.419553 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:08.420032 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:08.420043 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:08.422284 (+ 2241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
W20260928 13:09:08.422513 17758 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:08.417187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:35529 (local address 127.8.88.190:33141)
0928 13:09:08.417333 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:08.417335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:08.417343 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:09:08.417367 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:08.417367 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:08.417378 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:08.417436 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:08.417757 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:08.418249 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:08.419105 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:08.419322 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:08.419398 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:08.422419 (+ 3021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:35529: BlockingRecv error: recv got EOF from 127.8.88.129:35529 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
Sep 28 13:09:08 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:08 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:08.576809 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:08.571392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:52275)
0928 13:09:08.571559 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:08.571579 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:08.571643 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:08.571774 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:08.571774 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:08.571935 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:08.571982 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:08.571984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:08.572832 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:08.572833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:08.573467 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:08.573469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:08.574102 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:08.574103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:08.574158 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:08.574752 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:08.574757 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:08.576747 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
W20260928 13:09:08.577128 17758 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:08.571507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:52275 (local address 127.8.88.190:33141)
0928 13:09:08.571582 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:08.571584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:08.571643 (+ 59us) server_negotiation.cc:407] Connection header received
0928 13:09:08.571700 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:08.571700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:08.571712 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:08.571759 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:08.572078 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:08.572734 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:08.573566 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:08.573827 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:08.573887 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:08.577066 (+ 3179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:52275: BlockingRecv error: recv got EOF from 127.8.88.130:52275 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 13:09:08 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:08 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:08.714001 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:08.708596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:45121)
0928 13:09:08.708906 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:08.708918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:08.709013 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:08.709241 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:08.709243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:08.709432 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:08.709500 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:08.709502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:08.710256 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:08.710257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:08.710804 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:08.710806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:08.711194 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:08.711195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:08.711262 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:08.711795 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:08.711800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:08.713903 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":184,"threads_started":1}
W20260928 13:09:08.714063 17758 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:08.708719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:45121 (local address 127.8.88.190:33141)
0928 13:09:08.708838 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:08.708840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:08.708995 (+ 155us) server_negotiation.cc:407] Connection header received
0928 13:09:08.709094 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:08.709095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:08.709110 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:08.709216 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:08.709584 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:08.710149 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:08.710893 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:08.711093 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:08.711191 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:08.713995 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:45121: BlockingRecv error: recv got EOF from 127.8.88.131:45121 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 13:09:09 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:09 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:09.432988 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:09.422947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:57545)
0928 13:09:09.423107 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:09.423118 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:09.423178 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:09.423330 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:09.423331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:09.423558 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:09.423607 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:09.423609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:09.424331 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:09.424332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:09.424999 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:09.425002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:09.425424 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:09.425425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:09.425476 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:09.430413 (+ 4937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:09.430421 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:09.432842 (+ 2421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
W20260928 13:09:09.433208 17762 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:09.423038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:57545 (local address 127.8.88.190:33141)
0928 13:09:09.423208 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:09.423210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:09.423219 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:09:09.423243 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:09.423243 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:09.423253 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:09.423313 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:09.423694 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:09.424220 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:09.425102 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:09.425327 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:09.425392 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:09.433138 (+ 7746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:57545: BlockingRecv error: recv got EOF from 127.8.88.129:57545 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":114,"threads_started":1}
Sep 28 13:09:09 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:09 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:09.583101 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:09.577355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:37093)
0928 13:09:09.577606 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:09.577621 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:09.577728 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:09.577929 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:09.577930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:09.578180 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:09.578228 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:09.578231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:09.579008 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:09.579010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:09.579611 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:09.579613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:09.580068 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:09.580069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:09.580138 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:09.580816 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:09.580821 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:09.582991 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":122,"threads_started":1}
W20260928 13:09:09.583165 17762 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:09.577480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:37093 (local address 127.8.88.190:33141)
0928 13:09:09.577608 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:09.577611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:09.577708 (+ 97us) server_negotiation.cc:407] Connection header received
0928 13:09:09.577819 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:09.577820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:09.577835 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:09.577923 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:09.578331 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:09.578904 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:09.579716 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:09.579941 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:09.580047 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:09.583095 (+ 3048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:37093: BlockingRecv error: recv got EOF from 127.8.88.130:37093 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 13:09:09 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:09 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:09.719782 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:09.714672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:35597)
0928 13:09:09.714832 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:09.714846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:09.714915 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:09.715054 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:09.715054 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:09.715257 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:09.715308 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:09.715310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:09.716074 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:09.716075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:09.716639 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:09.716641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:09.717175 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:09.717176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:09.717224 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:09.717752 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:09.717758 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:09.719719 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
W20260928 13:09:09.719987 17762 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:09.714773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:35597 (local address 127.8.88.190:33141)
0928 13:09:09.714829 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:09.714831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:09.714902 (+ 71us) server_negotiation.cc:407] Connection header received
0928 13:09:09.714963 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:09.714964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:09.714975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:09.715034 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:09.715401 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:09.715969 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:09.716750 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:09.717033 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:09.717113 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:09.719906 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:35597: BlockingRecv error: recv got EOF from 127.8.88.131:35597 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 13:09:10 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:10 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:10.439064 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:10.433644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:39047)
0928 13:09:10.433813 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:10.433825 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:10.433889 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:10.434134 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:10.434135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:10.434313 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:10.434363 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:10.434365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:10.435117 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:10.435118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:10.435728 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:10.435730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:10.436123 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:10.436124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:10.436167 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:10.436812 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:10.436829 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:10.439003 (+ 2174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
W20260928 13:09:10.439218 17766 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:10.433766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:39047 (local address 127.8.88.190:33141)
0928 13:09:10.433949 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:10.433951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:10.433961 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:09:10.433988 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:10.433989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:10.433999 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:10.434084 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:10.434461 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:10.435013 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:10.435830 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:10.436038 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:10.436105 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:10.439123 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:39047: BlockingRecv error: recv got EOF from 127.8.88.129:39047 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":113,"threads_started":1}
Sep 28 13:09:10 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:10 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:10.590054 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:10.583831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:39483)
0928 13:09:10.584055 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:10.584075 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:10.584153 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:10.584304 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:10.584305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:10.584472 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:10.584526 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:10.584528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:10.585647 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:10.585648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:10.586259 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:10.586262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:10.586822 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:10.586824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:10.586873 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:10.587395 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:10.587400 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:10.589978 (+ 2578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
W20260928 13:09:10.590361 17766 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:10.583996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:39483 (local address 127.8.88.190:33141)
0928 13:09:10.584122 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:10.584124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:10.584141 (+ 17us) server_negotiation.cc:407] Connection header received
0928 13:09:10.584210 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:10.584211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:10.584223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:10.584282 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:10.584632 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:10.585558 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:10.586366 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:10.586700 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:10.586777 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:10.590311 (+ 3534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:39483: BlockingRecv error: recv got EOF from 127.8.88.130:39483 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 13:09:10 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:10 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:10.726118 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:10.720444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:37185)
0928 13:09:10.720631 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:10.720645 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:10.720722 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:10.720932 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:10.720933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:10.721140 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:10.721190 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:10.721192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:10.722093 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:10.722095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:10.722806 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:10.722809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:10.723224 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:10.723225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:10.723277 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:10.723837 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:10.723842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:10.726058 (+ 2216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
W20260928 13:09:10.726398 17766 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:10.720624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:37185 (local address 127.8.88.190:33141)
0928 13:09:10.720765 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:10.720767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:10.720776 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:09:10.720796 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:10.720797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:10.720830 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:10.720921 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:10.721316 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:10.721930 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:10.722897 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:10.723127 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:10.723189 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:10.726315 (+ 3126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:37185: BlockingRecv error: recv got EOF from 127.8.88.131:37185 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 28 13:09:11 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:11 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:11.444563 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:11.439727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:56967)
0928 13:09:11.439925 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:11.439941 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:11.440042 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:11.440218 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:11.440220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:11.440400 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:11.440508 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:11.440510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:11.441191 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:11.441192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:11.441794 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:11.441796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:11.442124 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:11.442125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:11.442170 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:11.442644 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:11.442648 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:11.444500 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":81,"threads_started":1}
W20260928 13:09:11.444733 17770 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:11.439807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:56967 (local address 127.8.88.190:33141)
0928 13:09:11.440043 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:11.440045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:11.440080 (+ 35us) server_negotiation.cc:407] Connection header received
0928 13:09:11.440104 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:11.440105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:11.440115 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:11.440171 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:11.440587 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:11.441123 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:11.441870 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:11.442061 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:11.442111 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:11.444624 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:56967: BlockingRecv error: recv got EOF from 127.8.88.129:56967 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":111,"threads_started":1}
Sep 28 13:09:11 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:11 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:11.595557 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:11.590649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:51885)
0928 13:09:11.590799 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:11.590815 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:11.590866 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:11.591013 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:11.591014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:11.591180 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:11.591225 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:11.591226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:11.591914 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:11.591915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:11.592539 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:11.592541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:11.592993 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:11.592994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:11.593065 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:11.593529 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:11.593533 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:11.595432 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
W20260928 13:09:11.595665 17770 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:11.590778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:51885 (local address 127.8.88.190:33141)
0928 13:09:11.590846 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:11.590848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:11.590860 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:09:11.590922 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:11.590923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:11.590933 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:11.590996 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:11.591298 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:11.591829 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:11.592643 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:11.592854 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:11.592987 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:11.595602 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:51885: BlockingRecv error: recv got EOF from 127.8.88.130:51885 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 13:09:11 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:11 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:11.731606 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:11.726636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:35151)
0928 13:09:11.726841 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:11.726853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:11.726949 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:11.727110 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:11.727111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:11.727295 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:11.727339 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:11.727340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:11.728063 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:11.728065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:11.728618 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:11.728620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:11.729042 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:11.729044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:11.729100 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:11.729597 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:11.729602 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:11.731491 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
W20260928 13:09:11.731642 17770 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:11.726731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:35151 (local address 127.8.88.190:33141)
0928 13:09:11.726811 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:11.726813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:11.726929 (+ 116us) server_negotiation.cc:407] Connection header received
0928 13:09:11.727004 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:11.727005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:11.727020 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:11.727088 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:11.727470 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:11.727977 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:11.728715 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:11.728968 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:11.729045 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:11.731575 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:35151: BlockingRecv error: recv got EOF from 127.8.88.131:35151 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 13:09:12 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:12 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:12.450989 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:12.445206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:43725)
0928 13:09:12.445435 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:12.445447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:12.445528 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:12.445785 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:12.445786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:12.445991 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:12.446041 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:12.446043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:12.446828 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:12.446829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:12.447385 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:12.447387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:12.447822 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:12.447823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:12.447871 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:12.448485 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:12.448490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:12.450876 (+ 2386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
W20260928 13:09:12.451051 17774 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:12.445407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:43725 (local address 127.8.88.190:33141)
0928 13:09:12.445631 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:12.445633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:12.445642 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:09:12.445667 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:12.445668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:12.445680 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:12.445744 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:12.446147 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:12.446738 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:12.447476 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:12.447727 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:12.447788 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:12.450980 (+ 3192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:43725: BlockingRecv error: recv got EOF from 127.8.88.129:43725 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":166,"threads_started":1}
Sep 28 13:09:12 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:12 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:12.601361 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:12.596093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:45323)
0928 13:09:12.596352 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:12.596365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:12.596428 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:12.596598 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:12.596599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:12.596763 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:12.596810 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:12.596812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:12.597589 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:12.597590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:12.598144 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:12.598146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:12.598530 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:12.598531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:12.598574 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:12.599145 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:12.599150 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:12.601249 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":74,"threads_started":1}
W20260928 13:09:12.601423 17774 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:12.596307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:45323 (local address 127.8.88.190:33141)
0928 13:09:12.596432 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:12.596434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:12.596444 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:09:12.596510 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:12.596510 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:12.596522 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:12.596573 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:12.596919 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:12.597495 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:12.598240 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:12.598443 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:12.598518 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:12.601356 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:45323: BlockingRecv error: recv got EOF from 127.8.88.130:45323 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 28 13:09:12 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:12 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:12.737713 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:12.732314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:33693)
0928 13:09:12.732607 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:12.732621 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:12.732715 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:12.732987 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:12.732988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:12.733172 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:12.733218 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:12.733220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:12.733938 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:12.733939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:12.734482 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:12.734484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:12.734867 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:12.734869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:12.734935 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:12.735466 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:12.735471 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:12.737608 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":160,"threads_started":1}
W20260928 13:09:12.737790 17774 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:12.732488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:33693 (local address 127.8.88.190:33141)
0928 13:09:12.732593 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:12.732596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:12.732694 (+ 98us) server_negotiation.cc:407] Connection header received
0928 13:09:12.732791 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:12.732792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:12.732848 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:12.732965 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:12.733301 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:12.733837 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:12.734581 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:12.734782 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:12.734870 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:12.737722 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:33693: BlockingRecv error: recv got EOF from 127.8.88.131:33693 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 13:09:13 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:13 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:13.457058 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:13.451735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:60833)
0928 13:09:13.451931 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:13.451944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:13.452026 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:13.452277 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:13.452278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:13.452534 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:13.452594 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:13.452596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:13.453391 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:13.453392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:13.453958 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:13.453960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:13.454356 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:13.454357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:13.454401 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:13.454917 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:13.454921 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:13.456994 (+ 2073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":133,"threads_started":1}
W20260928 13:09:13.457269 17778 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:13.451947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:60833 (local address 127.8.88.190:33141)
0928 13:09:13.452066 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:13.452068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:13.452113 (+ 45us) server_negotiation.cc:407] Connection header received
0928 13:09:13.452150 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:13.452150 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:13.452162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:13.452261 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:13.452680 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:13.453308 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:13.454059 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:13.454276 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:13.454338 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:13.457167 (+ 2829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:60833: BlockingRecv error: recv got EOF from 127.8.88.129:60833 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
Sep 28 13:09:13 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:13 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:13.607928 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:13.601871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:51235)
0928 13:09:13.602040 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:13.602051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:13.602109 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:13.602245 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:13.602246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:13.602469 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:13.602517 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:13.602519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:13.603444 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:13.603445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:13.603999 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:13.604001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:13.604480 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:13.604481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:13.604543 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:13.605205 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:13.605209 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:13.607846 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
W20260928 13:09:13.608083 17778 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:13.601991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:51235 (local address 127.8.88.190:33141)
0928 13:09:13.602059 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:13.602066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:13.602093 (+ 27us) server_negotiation.cc:407] Connection header received
0928 13:09:13.602163 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:13.602164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:13.602175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:13.602236 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:13.602586 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:13.603340 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:13.604074 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:13.604372 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:13.604441 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:13.607996 (+ 3555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:51235: BlockingRecv error: recv got EOF from 127.8.88.130:51235 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 13:09:13 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:13 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:13.743417 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:13.738360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:42425)
0928 13:09:13.738523 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:13.738540 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:13.738600 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:13.738820 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:13.738821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:13.738995 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:13.739044 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:13.739045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:13.739761 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:13.739762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:13.740389 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:13.740391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:13.740744 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:13.740745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:13.740816 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:13.741390 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:13.741395 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:13.743325 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":71,"threads_started":1}
W20260928 13:09:13.743562 17778 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:13.738478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:42425 (local address 127.8.88.190:33141)
0928 13:09:13.738558 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:13.738559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:13.738588 (+ 29us) server_negotiation.cc:407] Connection header received
0928 13:09:13.738714 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:13.738714 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:13.738727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:13.738771 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:13.739116 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:13.739666 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:13.740481 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:13.740668 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:13.740724 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:13.743488 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:42425: BlockingRecv error: recv got EOF from 127.8.88.131:42425 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 13:09:14 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:14 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:14.471438 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:14.457659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:42825)
0928 13:09:14.457846 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:14.457858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:14.457919 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:14.458114 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:14.458116 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:14.458462 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:14.458530 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:14.458533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:14.459299 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:14.459300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:14.459880 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:14.459882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:14.460316 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:14.460317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:14.460374 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:14.468988 (+ 8614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:14.468995 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:14.471375 (+ 2380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":105,"threads_started":1}
W20260928 13:09:14.471717 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:14.457797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:42825 (local address 127.8.88.190:33141)
0928 13:09:14.457931 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:14.457934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:14.457942 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:09:14.457972 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:14.457973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:14.457986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:14.458097 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:14.458629 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:14.459187 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:14.459993 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:14.460208 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:14.460273 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:14.471576 (+ 11303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:42825: BlockingRecv error: recv got EOF from 127.8.88.129:42825 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
Sep 28 13:09:14 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:14 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:14.613760 17783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:14.608481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:36491)
0928 13:09:14.608665 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:14.608685 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:14.608746 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:14.608942 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:14.608943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:14.609106 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:14.609152 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:14.609154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:14.609859 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:14.609860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:14.610440 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:14.610442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:14.610852 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:14.610853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:14.610902 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:14.611456 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:14.611461 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:14.613682 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":133,"threads_started":1}
W20260928 13:09:14.613914 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:14.608603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:36491 (local address 127.8.88.190:33141)
0928 13:09:14.608685 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:14.608687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:14.608731 (+ 44us) server_negotiation.cc:407] Connection header received
0928 13:09:14.608808 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:14.608809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:14.608820 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:14.608928 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:14.609228 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:14.609764 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:14.610542 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:14.610748 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:14.610815 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:14.613823 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:36491: BlockingRecv error: recv got EOF from 127.8.88.130:36491 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 13:09:14 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:14 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.131@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:14.749892 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:14.743983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.131:38385)
0928 13:09:14.744189 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:14.744200 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:14.744261 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:14.744499 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:14.744500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:14.744685 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:14.744735 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:14.744737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:14.745459 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:14.745461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:14.746058 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:14.746061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:14.746551 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:14.746552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:14.746613 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:14.747237 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:14.747242 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:14.749817 (+ 2575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":127,"threads_started":1}
W20260928 13:09:14.750033 17782 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:14.744097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:38385 (local address 127.8.88.190:33141)
0928 13:09:14.744174 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:14.744175 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:14.744262 (+ 87us) server_negotiation.cc:407] Connection header received
0928 13:09:14.744372 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:14.744373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:14.744386 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:14.744433 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:14.744812 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:14.745369 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:14.746217 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:14.746421 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:14.746480 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:14.749945 (+ 3465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.131:38385: BlockingRecv error: recv got EOF from 127.8.88.131:38385 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 13:09:15 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:15 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.129@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:15.477502 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:15.472096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.129:54693)
0928 13:09:15.472395 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:15.472407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:15.472483 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:15.472662 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:15.472664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:15.472830 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:15.472953 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:15.472960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:15.473775 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:15.473776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:15.474335 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:15.474337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:15.474703 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:15.474703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:15.474750 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:15.475321 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:15.475326 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:15.477443 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":124,"threads_started":1}
W20260928 13:09:15.477612 17786 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:15.472202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:54693 (local address 127.8.88.190:33141)
0928 13:09:15.472319 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:15.472321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:15.472499 (+ 178us) server_negotiation.cc:407] Connection header received
0928 13:09:15.472534 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:15.472534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:15.472548 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:15.472616 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:15.473081 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:15.473683 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:15.474421 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:15.474623 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:15.474699 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:15.477532 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.129:54693: BlockingRecv error: recv got EOF from 127.8.88.129:54693 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":57,"threads_started":1}
Sep 28 13:09:15 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
Sep 28 13:09:15 dist-test-slave-p1lc krb5kdc[16727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.88.130@KRBTEST.COM for oryx/127.8.88.190@KRBTEST.COM, Server not found in Kerberos database
I20260928 13:09:15.619653 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:15.614390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:33141 (local address 127.8.88.130:53101)
0928 13:09:15.614580 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:15.614591 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:15.614644 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:15.614883 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:15.614884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:15.615045 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:15.615093 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:15.615094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:15.615821 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:15.615821 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:15.616438 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:15.616441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:15.616842 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:15.616842 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:15.616952 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:15.617466 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:15.617471 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:15.619548 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.88.190:33141: Server oryx/127.8.88.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
W20260928 13:09:15.619711 17786 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:15.614485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:53101 (local address 127.8.88.190:33141)
0928 13:09:15.614613 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:15.614614 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:15.614627 (+ 13us) server_negotiation.cc:407] Connection header received
0928 13:09:15.614754 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:15.614755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:15.614769 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:15.614839 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:15.615179 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:15.615742 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:15.616532 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:15.616768 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:15.616831 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:15.619663 (+ 2832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.88.130:53101: BlockingRecv error: recv got EOF from 127.8.88.130:53101 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 13:09:15.667258 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 17310
I20260928 13:09:15.672006 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 17462
I20260928 13:09:15.677217 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 17596
I20260928 13:09:15.681777 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 17240
2026-09-28T13:09:15Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17647 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T13:09:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:09:15Z Disabled control of system clock
I20260928 13:09:15.711231 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:37369
--webserver_interface=127.8.88.190
--webserver_port=0
--builtin_ntp_servers=127.8.88.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:37369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 13:09:15.796669 17791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:15.796933 17791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:15.796957 17791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:15.798400 17791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:09:15.798467 17791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:15.798481 17791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:15.798494 17791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:09:15.798506 17791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:09:15.800213 17791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:37369
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:37369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:15.800532 17791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:15.800901 17791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:15.806555 17796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:15.806567 17791 server_base.cc:1034] running on GCE node
W20260928 13:09:15.806548 17797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:15.806548 17799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:15.807050 17791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:15.807451 17791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:15.808614 17791 hybrid_clock.cc:648] HybridClock initialized: now 1790600955808590 us; error 43 us; skew 500 ppm
I20260928 13:09:15.810122 17791 webserver.cc:460] Webserver started at http://127.8.88.190:33587/ using document root <none> and password file <none>
I20260928 13:09:15.810424 17791 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:15.810500 17791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:15.810652 17791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:09:15.811830 17791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "7b5b1f2470d84800a2bda23414b41923"
format_stamp: "Formatted at 2026-09-28 13:09:15 on dist-test-slave-p1lc"
server_key: "eee6a1ac43473eaa68837f8627b552a9"
server_key_iv: "f9a848b95b046bf907797ff67ec2e98c"
server_key_version: "encryptionkey@0"
I20260928 13:09:15.812219 17791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "7b5b1f2470d84800a2bda23414b41923"
format_stamp: "Formatted at 2026-09-28 13:09:15 on dist-test-slave-p1lc"
server_key: "eee6a1ac43473eaa68837f8627b552a9"
server_key_iv: "f9a848b95b046bf907797ff67ec2e98c"
server_key_version: "encryptionkey@0"
I20260928 13:09:15.813732 17791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 13:09:15.814637 17805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:15.814862 17791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:09:15.814972 17791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "7b5b1f2470d84800a2bda23414b41923"
format_stamp: "Formatted at 2026-09-28 13:09:15 on dist-test-slave-p1lc"
server_key: "eee6a1ac43473eaa68837f8627b552a9"
server_key_iv: "f9a848b95b046bf907797ff67ec2e98c"
server_key_version: "encryptionkey@0"
I20260928 13:09:15.815055 17791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:15.840440 17791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:15.840958 17791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:15.841131 17791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:15.845304 17791 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:37369
I20260928 13:09:15.845342 17857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:37369 every 8 connection(s)
I20260928 13:09:15.845739 17791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:09:15.846364 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 17791
I20260928 13:09:15.846475 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:09:15.846443 17858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:09:15.846688 8546 external_mini_cluster.cc:1428] Setting key c4cc8b86696d148042a955ac0d9f7883
I20260928 13:09:15.850335 17858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923: Bootstrap starting.
I20260928 13:09:15.851104 17858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923: Neither blocks nor log segments found. Creating new log.
I20260928 13:09:15.851432 17858 log.cc:824] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923: Log is configured to *not* fsync() on all Append() calls
I20260928 13:09:15.852306 17858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923: No bootstrap required, opened a new log
I20260928 13:09:15.852559 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:15.847263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.8.88.190:37369)
0928 13:09:15.847454 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:15.847461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:15.847487 (+ 26us) server_negotiation.cc:407] Connection header received
0928 13:09:15.847931 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:15.847937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:15.848001 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:15.848117 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:15.848359 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:15.849286 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:15.850460 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:15.850907 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:15.851007 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:15.851799 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:15.851804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:15.851804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:09:15.852023 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:15.852060 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:09:15.852072 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:15.852126 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:15.852239 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 13:09:15.852325 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260928 13:09:15.854297 17858 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5b1f2470d84800a2bda23414b41923" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37369 } }
I20260928 13:09:15.854485 17858 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:09:15.854537 17858 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5b1f2470d84800a2bda23414b41923, State: Initialized, Role: FOLLOWER
I20260928 13:09:15.854696 17858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5b1f2470d84800a2bda23414b41923" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37369 } }
I20260928 13:09:15.854827 17858 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:09:15.854866 17858 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:09:15.854923 17858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:09:15.855537 17858 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5b1f2470d84800a2bda23414b41923" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37369 } }
I20260928 13:09:15.855633 17858 leader_election.cc:302] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b5b1f2470d84800a2bda23414b41923; no voters:
I20260928 13:09:15.855934 17858 leader_election.cc:288] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:09:15.856022 17863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:09:15.856192 17863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [term 1 LEADER]: Becoming Leader. State: Replica: 7b5b1f2470d84800a2bda23414b41923, State: Running, Role: LEADER
I20260928 13:09:15.856334 17858 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:09:15.856412 17863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5b1f2470d84800a2bda23414b41923" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37369 } }
I20260928 13:09:15.856817 17865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b5b1f2470d84800a2bda23414b41923. Latest consensus state: current_term: 1 leader_uuid: "7b5b1f2470d84800a2bda23414b41923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5b1f2470d84800a2bda23414b41923" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37369 } } }
I20260928 13:09:15.856952 17865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [sys.catalog]: This master's current role is: LEADER
I20260928 13:09:15.857179 17872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:09:15.856797 17864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b5b1f2470d84800a2bda23414b41923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b5b1f2470d84800a2bda23414b41923" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 37369 } } }
I20260928 13:09:15.857534 17864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923 [sys.catalog]: This master's current role is: LEADER
I20260928 13:09:15.857762 17872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:09:15.859556 17872 catalog_manager.cc:1347] Generated new cluster ID: 40c7650b125041b1ad9afbf41892c225
I20260928 13:09:15.859619 17872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:09:15.882792 17872 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:09:15.883659 17872 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:09:15.890964 17872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b5b1f2470d84800a2bda23414b41923: Generated new TSK 0
I20260928 13:09:15.891214 17872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:09:15.896028 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:37369
--builtin_ntp_servers=127.8.88.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 13:09:15.982060 17882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:15.982350 17882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:15.982388 17882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:15.983666 17882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:15.983738 17882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:15.983783 17882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:09:15.985291 17882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:37369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:15.985563 17882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:15.985888 17882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:15.991744 17890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:15.991724 17887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:15.991945 17888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:15.991856 17882 server_base.cc:1034] running on GCE node
I20260928 13:09:15.992385 17882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:15.992661 17882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:15.993824 17882 hybrid_clock.cc:648] HybridClock initialized: now 1790600955993810 us; error 30 us; skew 500 ppm
I20260928 13:09:15.995493 17882 webserver.cc:460] Webserver started at http://127.8.88.129:45655/ using document root <none> and password file <none>
I20260928 13:09:15.995761 17882 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:15.995822 17882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:15.995946 17882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:09:15.997383 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "171f49a39cbf4cc5be4f82733576546e"
format_stamp: "Formatted at 2026-09-28 13:09:15 on dist-test-slave-p1lc"
server_key: "fcc13377477f8100cba34e9a7a1f4aa4"
server_key_iv: "aeee28f7a764fdf727c712125467f457"
server_key_version: "encryptionkey@0"
I20260928 13:09:15.997809 17882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "171f49a39cbf4cc5be4f82733576546e"
format_stamp: "Formatted at 2026-09-28 13:09:15 on dist-test-slave-p1lc"
server_key: "fcc13377477f8100cba34e9a7a1f4aa4"
server_key_iv: "aeee28f7a764fdf727c712125467f457"
server_key_version: "encryptionkey@0"
I20260928 13:09:15.999701 17882 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 13:09:16.000602 17896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.000855 17882 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:09:16.001025 17882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "171f49a39cbf4cc5be4f82733576546e"
format_stamp: "Formatted at 2026-09-28 13:09:15 on dist-test-slave-p1lc"
server_key: "fcc13377477f8100cba34e9a7a1f4aa4"
server_key_iv: "aeee28f7a764fdf727c712125467f457"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.001194 17882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:16.017710 17882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:16.018154 17882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:16.018332 17882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:16.018620 17882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:09:16.019018 17882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:09:16.019078 17882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.019114 17882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:09:16.019166 17882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.025591 17882 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:40139
I20260928 13:09:16.025631 18009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:40139 every 8 connection(s)
I20260928 13:09:16.025985 17882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:09:16.031958 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 17882
I20260928 13:09:16.032089 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:09:16.032181 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:16.026816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:44247 (local address 127.8.88.190:37369)
0928 13:09:16.026902 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:16.026905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:16.027142 (+ 237us) server_negotiation.cc:407] Connection header received
0928 13:09:16.027957 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:16.027958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:16.027977 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:16.028045 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:16.028834 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:16.029434 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:16.030480 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:16.030689 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:16.030781 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:16.031828 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:16.031830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:16.031831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:09:16.031838 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:16.031861 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:09:16.031865 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:16.031929 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:16.032045 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 13:09:16.032080 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 13:09:16.032356 8546 external_mini_cluster.cc:1428] Setting key d6eb195d6d55ab2ae18964b05035608e
I20260928 13:09:16.032534 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:16.026817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:37369 (local address 127.8.88.129:44247)
0928 13:09:16.027010 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:16.027046 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:16.027644 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:16.028192 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:16.028196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:16.028206 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:16.028725 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:16.028730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:16.029544 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:16.029545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:16.030377 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:16.030379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:16.030796 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:16.030797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:16.030953 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:16.031601 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:09:16.031601 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:16.031647 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:09:16.031674 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:09:16.031943 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:09:16.031951 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:09:16.031955 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:09:16.032049 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 13:09:16.032172 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260928 13:09:16.033200 18010 heartbeater.cc:347] Connected to a master server at 127.8.88.190:37369
I20260928 13:09:16.033317 18010 heartbeater.cc:464] Registering TS with master...
I20260928 13:09:16.033474 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:37369
--builtin_ntp_servers=127.8.88.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 13:09:16.033548 18010 heartbeater.cc:511] Master 127.8.88.190:37369 requested a full tablet report, sending...
I20260928 13:09:16.034289 17822 ts_manager.cc:194] Registered new tserver with Master: 171f49a39cbf4cc5be4f82733576546e (127.8.88.129:40139)
I20260928 13:09:16.034986 17822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.88.129:44247
W20260928 13:09:16.121774 18013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:16.122020 18013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:16.122046 18013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:16.123373 18013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:16.123430 18013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:16.123458 18013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:09:16.124943 18013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:37369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:16.125226 18013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:16.125538 18013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:16.131134 18021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:16.131146 18018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:16.131134 18019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:16.131465 18013 server_base.cc:1034] running on GCE node
I20260928 13:09:16.131695 18013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:16.131929 18013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:16.133167 18013 hybrid_clock.cc:648] HybridClock initialized: now 1790600956133130 us; error 54 us; skew 500 ppm
I20260928 13:09:16.134666 18013 webserver.cc:460] Webserver started at http://127.8.88.130:37643/ using document root <none> and password file <none>
I20260928 13:09:16.134902 18013 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:16.134953 18013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:16.135076 18013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:09:16.136286 18013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "783c65e5ded04c959ca61949b0b4b677"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "289218d1900082f5b70a750063e7f401"
server_key_iv: "bbe4e1060dd80814f18cea1e1166144f"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.136613 18013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "783c65e5ded04c959ca61949b0b4b677"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "289218d1900082f5b70a750063e7f401"
server_key_iv: "bbe4e1060dd80814f18cea1e1166144f"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.138077 18013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:09:16.138912 18027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.139106 18013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:09:16.139215 18013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "783c65e5ded04c959ca61949b0b4b677"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "289218d1900082f5b70a750063e7f401"
server_key_iv: "bbe4e1060dd80814f18cea1e1166144f"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.139317 18013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:16.147918 18013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:16.148293 18013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:16.148459 18013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:16.148706 18013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:09:16.149164 18013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:09:16.149221 18013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.149264 18013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:09:16.149292 18013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.155292 18013 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:40961
I20260928 13:09:16.155341 18140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:40961 every 8 connection(s)
I20260928 13:09:16.155732 18013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:09:16.159435 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 18013
I20260928 13:09:16.159554 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:09:16.159785 8546 external_mini_cluster.cc:1428] Setting key 02b832fbba2aa8df9d205f2a49cdde2b
I20260928 13:09:16.161027 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:37369
--builtin_ntp_servers=127.8.88.148:40945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 13:09:16.163057 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:16.156622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:35473 (local address 127.8.88.190:37369)
0928 13:09:16.156688 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:16.156691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:16.156962 (+ 271us) server_negotiation.cc:407] Connection header received
0928 13:09:16.157627 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:16.157628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:16.157648 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:16.157712 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:16.158429 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:16.158952 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:16.160478 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:16.160792 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:16.160916 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:16.162727 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:16.162729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:16.162729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:09:16.162735 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:16.162753 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:09:16.162757 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:16.162835 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:16.162939 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 13:09:16.162973 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 13:09:16.163378 18143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:16.156613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:37369 (local address 127.8.88.130:35473)
0928 13:09:16.156813 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:16.156849 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:16.157452 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:16.157789 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:16.157793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:16.157805 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:16.158330 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:16.158335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:16.159062 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:16.159064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:16.160376 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:16.160379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:16.160978 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:16.160979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:16.161340 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:16.162439 (+ 1099us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:09:16.162439 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:16.162473 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:09:16.162503 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:09:16.162835 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:09:16.162843 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:09:16.162847 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:09:16.162944 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 13:09:16.163082 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260928 13:09:16.163941 18141 heartbeater.cc:347] Connected to a master server at 127.8.88.190:37369
I20260928 13:09:16.164098 18141 heartbeater.cc:464] Registering TS with master...
I20260928 13:09:16.164305 18141 heartbeater.cc:511] Master 127.8.88.190:37369 requested a full tablet report, sending...
I20260928 13:09:16.164731 17822 ts_manager.cc:194] Registered new tserver with Master: 783c65e5ded04c959ca61949b0b4b677 (127.8.88.130:40961)
I20260928 13:09:16.165205 17822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.88.130:35473
W20260928 13:09:16.253943 18144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:16.254185 18144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:16.254209 18144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:16.255501 18144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:16.255564 18144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:16.255592 18144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:09:16.257764 18144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:37369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:16.258093 18144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:16.258443 18144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:16.264290 18152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:16.264298 18149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:16.264299 18150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:16.264396 18144 server_base.cc:1034] running on GCE node
I20260928 13:09:16.264727 18144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:16.265030 18144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:16.266254 18144 hybrid_clock.cc:648] HybridClock initialized: now 1790600956266192 us; error 32 us; skew 500 ppm
I20260928 13:09:16.268085 18144 webserver.cc:460] Webserver started at http://127.8.88.131:36879/ using document root <none> and password file <none>
I20260928 13:09:16.268381 18144 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:16.268432 18144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:16.268579 18144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:09:16.269797 18144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "d0c0da47908c4ff89d1f66670d32ba91"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "9756c294df13d8823b2a795fbd9a5b39"
server_key_iv: "12e87049ab1daf273edbe934655978e8"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.270187 18144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "d0c0da47908c4ff89d1f66670d32ba91"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "9756c294df13d8823b2a795fbd9a5b39"
server_key_iv: "12e87049ab1daf273edbe934655978e8"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.271621 18144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 13:09:16.272580 18158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.272959 18144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:09:16.273067 18144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "d0c0da47908c4ff89d1f66670d32ba91"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "9756c294df13d8823b2a795fbd9a5b39"
server_key_iv: "12e87049ab1daf273edbe934655978e8"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.273160 18144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:16.286644 18144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:16.286995 18144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:16.287158 18144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:16.287424 18144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:09:16.287808 18144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:09:16.287863 18144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.287900 18144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:09:16.287914 18144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.294035 18144 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:43105
I20260928 13:09:16.294078 18271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:43105 every 8 connection(s)
I20260928 13:09:16.294446 18144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:09:16.298329 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 18144
I20260928 13:09:16.298451 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:09:16.298717 8546 external_mini_cluster.cc:1428] Setting key bd7ce8bef539f2a81100537597b07113
I20260928 13:09:16.301069 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:16.295218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:55587 (local address 127.8.88.190:37369)
0928 13:09:16.295265 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:16.295268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:16.295534 (+ 266us) server_negotiation.cc:407] Connection header received
0928 13:09:16.296045 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:16.296047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:16.296061 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:16.296123 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:16.296836 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:16.297450 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:16.299255 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:16.299479 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:16.299632 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:16.300735 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:16.300737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:16.300737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:09:16.300742 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:16.300777 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:09:16.300781 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:16.300829 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:16.300979 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 13:09:16.301005 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:09:16.301332 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:16.295226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:37369 (local address 127.8.88.131:55587)
0928 13:09:16.295403 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:16.295437 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:16.295924 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:16.296181 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:16.296184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:16.296199 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:16.296721 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:16.296726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:16.297592 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:16.297593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:16.299158 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:16.299162 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:16.299657 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:16.299658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:16.299843 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:16.300575 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:09:16.300576 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:16.300610 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:09:16.300639 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:09:16.300831 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:09:16.300834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:09:16.300837 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:09:16.300955 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 13:09:16.301067 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260928 13:09:16.301826 18272 heartbeater.cc:347] Connected to a master server at 127.8.88.190:37369
I20260928 13:09:16.301919 18272 heartbeater.cc:464] Registering TS with master...
I20260928 13:09:16.302207 18272 heartbeater.cc:511] Master 127.8.88.190:37369 requested a full tablet report, sending...
I20260928 13:09:16.302675 17822 ts_manager.cc:194] Registered new tserver with Master: d0c0da47908c4ff89d1f66670d32ba91 (127.8.88.131:43105)
I20260928 13:09:16.303056 17822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.88.131:55587
I20260928 13:09:16.311249 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 13:09:16.313776 17861 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:16.312949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46702 (local address 127.8.88.190:37369)
0928 13:09:16.313009 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:16.313028 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:16.313373 (+ 345us) server_negotiation.cc:407] Connection header received
0928 13:09:16.313399 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:16.313400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:16.313410 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:16.313468 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:16.313683 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46702: BlockingRecv error: recv got EOF from 127.0.0.1:46702 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 13:09:16.313783 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 17882
I20260928 13:09:16.318675 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 18013
I20260928 13:09:16.323320 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 18144
I20260928 13:09:16.329300 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 17791
2026-09-28T13:09:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (646 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T13:09:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:09:16Z Disabled control of system clock
I20260928 13:09:16.357635 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46469
--webserver_interface=127.8.88.190
--webserver_port=0
--builtin_ntp_servers=127.8.88.148:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 13:09:16.442889 18283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:16.443142 18283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:16.443167 18283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:16.444442 18283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:09:16.444495 18283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:16.444512 18283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:16.444526 18283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:09:16.444538 18283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:09:16.446017 18283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:16.446395 18283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:16.446740 18283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:16.452304 18288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:16.452324 18289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:16.452313 18291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:16.452478 18283 server_base.cc:1034] running on GCE node
I20260928 13:09:16.452778 18283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:16.453179 18283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:16.454362 18283 hybrid_clock.cc:648] HybridClock initialized: now 1790600956454336 us; error 40 us; skew 500 ppm
I20260928 13:09:16.455885 18283 webserver.cc:460] Webserver started at http://127.8.88.190:35213/ using document root <none> and password file <none>
I20260928 13:09:16.456219 18283 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:16.456303 18283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:16.456465 18283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:09:16.457677 18283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "3726a8a10bf943508f44fd1ecb070362"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "4d03fbcb6c31374de42f6a7d11bfcc94"
server_key_iv: "38194982c18b4ce528c3154de8ec26f5"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.458066 18283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "3726a8a10bf943508f44fd1ecb070362"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "4d03fbcb6c31374de42f6a7d11bfcc94"
server_key_iv: "38194982c18b4ce528c3154de8ec26f5"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.459563 18283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:09:16.460464 18297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.460685 18283 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:09:16.460800 18283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "3726a8a10bf943508f44fd1ecb070362"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "4d03fbcb6c31374de42f6a7d11bfcc94"
server_key_iv: "38194982c18b4ce528c3154de8ec26f5"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.460911 18283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:16.464648 18283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:16.465029 18283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:16.465189 18283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:16.469480 18283 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:46469
I20260928 13:09:16.469517 18349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:46469 every 8 connection(s)
I20260928 13:09:16.469882 18283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:09:16.470569 18350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:09:16.472348 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 18283
I20260928 13:09:16.472424 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:09:16.472640 8546 external_mini_cluster.cc:1428] Setting key 6729d1e1461b1d67ce0540573b95e6be
I20260928 13:09:16.473976 18350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362: Bootstrap starting.
I20260928 13:09:16.474733 18350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362: Neither blocks nor log segments found. Creating new log.
I20260928 13:09:16.475085 18350 log.cc:824] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362: Log is configured to *not* fsync() on all Append() calls
I20260928 13:09:16.475432 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:16.473283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.8.88.190:46469)
0928 13:09:16.473508 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:16.473519 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:16.473559 (+ 40us) server_negotiation.cc:407] Connection header received
0928 13:09:16.474086 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:16.474095 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:16.474148 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:16.474316 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:16.474959 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:16.474963 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:16.474964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:09:16.474968 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:16.475022 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:09:16.475036 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:16.475117 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:16.475163 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 13:09:16.475229 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260928 13:09:16.475890 18350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362: No bootstrap required, opened a new log
I20260928 13:09:16.477368 18350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3726a8a10bf943508f44fd1ecb070362" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46469 } }
I20260928 13:09:16.477538 18350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:09:16.477595 18350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3726a8a10bf943508f44fd1ecb070362, State: Initialized, Role: FOLLOWER
I20260928 13:09:16.477764 18350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3726a8a10bf943508f44fd1ecb070362" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46469 } }
I20260928 13:09:16.477873 18350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:09:16.477918 18350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:09:16.477967 18350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:09:16.478808 18350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3726a8a10bf943508f44fd1ecb070362" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46469 } }
I20260928 13:09:16.478931 18350 leader_election.cc:302] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3726a8a10bf943508f44fd1ecb070362; no voters:
I20260928 13:09:16.479159 18350 leader_election.cc:288] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:09:16.479193 18355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:09:16.479394 18355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [term 1 LEADER]: Becoming Leader. State: Replica: 3726a8a10bf943508f44fd1ecb070362, State: Running, Role: LEADER
I20260928 13:09:16.479491 18355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3726a8a10bf943508f44fd1ecb070362" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46469 } }
I20260928 13:09:16.479554 18350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:09:16.479861 18356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3726a8a10bf943508f44fd1ecb070362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3726a8a10bf943508f44fd1ecb070362" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46469 } } }
I20260928 13:09:16.479977 18356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [sys.catalog]: This master's current role is: LEADER
I20260928 13:09:16.479875 18357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3726a8a10bf943508f44fd1ecb070362. Latest consensus state: current_term: 1 leader_uuid: "3726a8a10bf943508f44fd1ecb070362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3726a8a10bf943508f44fd1ecb070362" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46469 } } }
I20260928 13:09:16.480134 18357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362 [sys.catalog]: This master's current role is: LEADER
I20260928 13:09:16.480443 18361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:09:16.480947 18361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:09:16.482630 18361 catalog_manager.cc:1347] Generated new cluster ID: 1c1884b17b92485bb6173a534c8e8178
I20260928 13:09:16.482685 18361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:09:16.490408 18361 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:09:16.490595 18361 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:09:16.493180 18361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3726a8a10bf943508f44fd1ecb070362: Generated new TSK 0
I20260928 13:09:16.493382 18361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:09:16.495723 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46469
--builtin_ntp_servers=127.8.88.148:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 13:09:16.582104 18374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:16.582332 18374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:16.582360 18374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:16.583685 18374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:16.583734 18374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:16.583765 18374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:09:16.585309 18374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:16.585589 18374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:16.585935 18374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:16.593168 18379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:16.593168 18382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:16.593233 18380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:16.593228 18374 server_base.cc:1034] running on GCE node
I20260928 13:09:16.593762 18374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:16.594136 18374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:16.595340 18374 hybrid_clock.cc:648] HybridClock initialized: now 1790600956595298 us; error 55 us; skew 500 ppm
I20260928 13:09:16.597056 18374 webserver.cc:460] Webserver started at http://127.8.88.129:45961/ using document root <none> and password file <none>
I20260928 13:09:16.597371 18374 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:16.597437 18374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:16.597568 18374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:09:16.599088 18374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "3aa027cb336c492e80dada05baedb31d"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "0d2dd7a1ba2447ccd1b829e5894fc238"
server_key_iv: "a34f2a9b3065dc68b8ce8719b812ec1b"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.599520 18374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "3aa027cb336c492e80dada05baedb31d"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "0d2dd7a1ba2447ccd1b829e5894fc238"
server_key_iv: "a34f2a9b3065dc68b8ce8719b812ec1b"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.601368 18374 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260928 13:09:16.602558 18388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.602843 18374 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:09:16.602941 18374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "3aa027cb336c492e80dada05baedb31d"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "0d2dd7a1ba2447ccd1b829e5894fc238"
server_key_iv: "a34f2a9b3065dc68b8ce8719b812ec1b"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.603034 18374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:16.607977 18374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:16.608299 18374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:16.608451 18374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:16.608732 18374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:09:16.609163 18374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:09:16.609210 18374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.609241 18374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:09:16.609259 18374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.616552 18374 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:45279
I20260928 13:09:16.616662 18501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:45279 every 8 connection(s)
I20260928 13:09:16.617030 18374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:09:16.620368 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:16.617861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:42197 (local address 127.8.88.190:46469)
0928 13:09:16.617911 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:16.617913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:16.618178 (+ 265us) server_negotiation.cc:407] Connection header received
0928 13:09:16.618706 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:16.618707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:16.618714 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:16.618764 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:16.620062 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:16.620064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:16.620065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:09:16.620070 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:16.620086 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:09:16.620091 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:16.620178 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:16.620275 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 13:09:16.620291 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:09:16.620582 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:16.617813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46469 (local address 127.8.88.129:42197)
0928 13:09:16.618021 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:16.618061 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:16.618583 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:16.618848 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:16.618852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:16.618860 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:16.619894 (+ 1034us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:09:16.619894 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:16.619925 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:09:16.619957 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:09:16.620177 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:09:16.620184 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:09:16.620187 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:09:16.620287 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 13:09:16.620356 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260928 13:09:16.621179 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 18374
I20260928 13:09:16.621273 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:09:16.621325 18502 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46469
I20260928 13:09:16.621420 18502 heartbeater.cc:464] Registering TS with master...
I20260928 13:09:16.621483 8546 external_mini_cluster.cc:1428] Setting key 2707fd8b900e6de6fb9203cfa365e812
I20260928 13:09:16.621584 18502 heartbeater.cc:511] Master 127.8.88.190:46469 requested a full tablet report, sending...
I20260928 13:09:16.622120 18313 ts_manager.cc:194] Registered new tserver with Master: 3aa027cb336c492e80dada05baedb31d (127.8.88.129:45279)
I20260928 13:09:16.622572 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46469
--builtin_ntp_servers=127.8.88.148:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 13:09:16.706872 18505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:16.707185 18505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:16.707228 18505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:16.708559 18505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:16.708611 18505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:16.708639 18505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:09:16.710146 18505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:16.710422 18505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:16.710784 18505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:09:16.716377 18505 server_base.cc:1034] running on GCE node
W20260928 13:09:16.716370 18513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:16.716365 18511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:16.716827 18510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:16.717125 18505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:16.717428 18505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:16.718618 18505 hybrid_clock.cc:648] HybridClock initialized: now 1790600956718608 us; error 32 us; skew 500 ppm
I20260928 13:09:16.720084 18505 webserver.cc:460] Webserver started at http://127.8.88.130:41831/ using document root <none> and password file <none>
I20260928 13:09:16.720367 18505 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:16.720448 18505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:16.720602 18505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:09:16.721835 18505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "bedb58aaa6ca48cba1b8cdf8b8be3fcd"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "f153f6d843c2d755651d03346734484b"
server_key_iv: "74b40eec5e3a60ce47dc8bd2f7a5c205"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.722218 18505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "bedb58aaa6ca48cba1b8cdf8b8be3fcd"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "f153f6d843c2d755651d03346734484b"
server_key_iv: "74b40eec5e3a60ce47dc8bd2f7a5c205"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.723601 18505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:09:16.724567 18519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.724828 18505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:09:16.724961 18505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "bedb58aaa6ca48cba1b8cdf8b8be3fcd"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "f153f6d843c2d755651d03346734484b"
server_key_iv: "74b40eec5e3a60ce47dc8bd2f7a5c205"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.725044 18505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:16.728763 18505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:16.729087 18505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:16.729239 18505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:16.729457 18505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:09:16.729782 18505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:09:16.729836 18505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.729877 18505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:09:16.729904 18505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.735994 18505 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:43291
I20260928 13:09:16.736032 18632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:43291 every 8 connection(s)
I20260928 13:09:16.736426 18505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:09:16.737790 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 18505
I20260928 13:09:16.737915 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:09:16.738119 8546 external_mini_cluster.cc:1428] Setting key db79dcf269e8fd7f4f37291e4d1e6261
I20260928 13:09:16.739449 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46469
--builtin_ntp_servers=127.8.88.148:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 13:09:16.740090 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:16.737227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:35399 (local address 127.8.88.190:46469)
0928 13:09:16.737276 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:16.737278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:16.737658 (+ 380us) server_negotiation.cc:407] Connection header received
0928 13:09:16.738295 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:16.738297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:16.738303 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:16.738363 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:16.739837 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:16.739839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:16.739839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:09:16.739843 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:16.739858 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:09:16.739862 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:16.739916 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:16.740022 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 13:09:16.740034 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:09:16.740346 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:16.737188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46469 (local address 127.8.88.130:35399)
0928 13:09:16.737499 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:16.737534 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:16.738185 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:16.738439 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:16.738444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:16.738456 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:16.739618 (+ 1162us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:09:16.739618 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:16.739669 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:09:16.739709 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:09:16.739936 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:09:16.739942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:09:16.739945 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:09:16.740026 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 13:09:16.740097 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":84,"threads_started":1}
I20260928 13:09:16.741003 18633 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46469
I20260928 13:09:16.741140 18633 heartbeater.cc:464] Registering TS with master...
I20260928 13:09:16.741309 18633 heartbeater.cc:511] Master 127.8.88.190:46469 requested a full tablet report, sending...
I20260928 13:09:16.741744 18313 ts_manager.cc:194] Registered new tserver with Master: bedb58aaa6ca48cba1b8cdf8b8be3fcd (127.8.88.130:43291)
W20260928 13:09:16.824922 18636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:16.825213 18636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:16.825254 18636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:16.826582 18636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:16.826686 18636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:16.826740 18636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:09:16.828325 18636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:16.828634 18636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:16.829069 18636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:16.834808 18642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:16.834818 18641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:16.834830 18644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:16.834856 18636 server_base.cc:1034] running on GCE node
I20260928 13:09:16.835266 18636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:16.835589 18636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:16.836790 18636 hybrid_clock.cc:648] HybridClock initialized: now 1790600956836759 us; error 47 us; skew 500 ppm
I20260928 13:09:16.838376 18636 webserver.cc:460] Webserver started at http://127.8.88.131:44907/ using document root <none> and password file <none>
I20260928 13:09:16.838641 18636 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:16.838721 18636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:16.838878 18636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:09:16.839982 18636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "b5a5ae608d3d4c23ac76e3221dcda753"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "d9f01d47485f6b00000827f5eb634488"
server_key_iv: "b7f5bfd83375e1c04a0e6acd840af200"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.840382 18636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "b5a5ae608d3d4c23ac76e3221dcda753"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "d9f01d47485f6b00000827f5eb634488"
server_key_iv: "b7f5bfd83375e1c04a0e6acd840af200"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.841881 18636 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:09:16.842939 18650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.843166 18636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:09:16.843276 18636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "b5a5ae608d3d4c23ac76e3221dcda753"
format_stamp: "Formatted at 2026-09-28 13:09:16 on dist-test-slave-p1lc"
server_key: "d9f01d47485f6b00000827f5eb634488"
server_key_iv: "b7f5bfd83375e1c04a0e6acd840af200"
server_key_version: "encryptionkey@0"
I20260928 13:09:16.843355 18636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:16.847376 18636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:16.847689 18636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:16.847847 18636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:16.848086 18636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:09:16.848475 18636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:09:16.848526 18636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.848565 18636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:09:16.848595 18636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:16.855238 18636 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:43489
I20260928 13:09:16.855279 18763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:43489 every 8 connection(s)
I20260928 13:09:16.855669 18636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:09:16.858829 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:16.856511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:46881 (local address 127.8.88.190:46469)
0928 13:09:16.856622 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:16.856624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:16.856737 (+ 113us) server_negotiation.cc:407] Connection header received
0928 13:09:16.857279 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:16.857281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:16.857287 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:16.857353 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:16.858515 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:16.858516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:16.858517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:09:16.858521 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:16.858550 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:09:16.858554 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:16.858612 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:16.858750 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 13:09:16.858767 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 13:09:16.858975 18766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:16.856367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46469 (local address 127.8.88.131:46881)
0928 13:09:16.856600 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:16.856649 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:16.857175 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:16.857425 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:16.857429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:16.857437 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:16.858356 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:09:16.858357 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:16.858394 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:09:16.858423 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:09:16.858646 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:09:16.858651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:09:16.858653 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 13:09:16.858715 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 13:09:16.858767 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":135,"threads_started":1}
I20260928 13:09:16.859622 18764 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46469
I20260928 13:09:16.859701 18764 heartbeater.cc:464] Registering TS with master...
I20260928 13:09:16.859848 18764 heartbeater.cc:511] Master 127.8.88.190:46469 requested a full tablet report, sending...
I20260928 13:09:16.860308 18313 ts_manager.cc:194] Registered new tserver with Master: b5a5ae608d3d4c23ac76e3221dcda753 (127.8.88.131:43489)
I20260928 13:09:16.865221 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 18636
I20260928 13:09:16.865331 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:09:16.865535 8546 external_mini_cluster.cc:1428] Setting key f3da376d6275412a2a220ddfc1496ea2
I20260928 13:09:16.866626 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 13:09:16.869025 18353 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:09:16.868333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.8.88.190:46469)
0928 13:09:16.868428 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:16.868430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:16.868535 (+ 105us) server_negotiation.cc:407] Connection header received
0928 13:09:16.868613 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:16.868614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:16.868621 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:16.868673 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:16.868905 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44722: BlockingRecv error: recv got EOF from 127.0.0.1:44722 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 13:09:16.869864 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 18374
I20260928 13:09:16.874864 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 18505
I20260928 13:09:16.879066 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 18636
I20260928 13:09:16.883822 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 18283
2026-09-28T13:09:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (554 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:09:16 dist-test-slave-p1lc krb5kdc[18774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:09:16 dist-test-slave-p1lc krb5kdc[18774](info): set up 2 sockets
Sep 28 13:09:16 dist-test-slave-p1lc krb5kdc[18774](info): commencing operation
krb5kdc: starting...
W20260928 13:09:17.969399 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:09:18 dist-test-slave-p1lc krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:09:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:09:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:09:18.106083 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:36437
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:36437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:09:18.194900 18787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:18.195143 18787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:18.195170 18787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:18.196799 18787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:09:18.196936 18787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:18.196974 18787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:18.196995 18787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:09:18.197028 18787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:09:18.199052 18787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:36437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:36437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:18.199368 18787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:18.199677 18787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:18.205127 18792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:18.205255 18787 server_base.cc:1034] running on GCE node
W20260928 13:09:18.205366 18795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:18.205166 18793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:18.205778 18787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:18.206084 18787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:18.207247 18787 hybrid_clock.cc:648] HybridClock initialized: now 1790600958207229 us; error 45 us; skew 500 ppm
Sep 28 13:09:18 dist-test-slave-p1lc krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600958, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:09:18.210222 18787 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:09:18.210721 18787 webserver.cc:460] Webserver started at http://127.8.88.190:43261/ using document root <none> and password file <none>
I20260928 13:09:18.211041 18787 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:18.211123 18787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:18.211268 18787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:09:18.212692 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "1e2f579ffbff41ac990817dd63144212"
format_stamp: "Formatted at 2026-09-28 13:09:18 on dist-test-slave-p1lc"
server_key: "87feae5064a90df3fcc7d8f494684d26"
server_key_iv: "30e4a42001e15aa92b5e7345249c323d"
server_key_version: "encryptionkey@0"
I20260928 13:09:18.213126 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "1e2f579ffbff41ac990817dd63144212"
format_stamp: "Formatted at 2026-09-28 13:09:18 on dist-test-slave-p1lc"
server_key: "87feae5064a90df3fcc7d8f494684d26"
server_key_iv: "30e4a42001e15aa92b5e7345249c323d"
server_key_version: "encryptionkey@0"
I20260928 13:09:18.214629 18787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 13:09:18.215458 18802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:18.215801 18787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:09:18.215968 18787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "1e2f579ffbff41ac990817dd63144212"
format_stamp: "Formatted at 2026-09-28 13:09:18 on dist-test-slave-p1lc"
server_key: "87feae5064a90df3fcc7d8f494684d26"
server_key_iv: "30e4a42001e15aa92b5e7345249c323d"
server_key_version: "encryptionkey@0"
I20260928 13:09:18.216058 18787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:18.231729 18787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:18.234391 18787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:18.234603 18787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:18.239240 18787 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:36437
I20260928 13:09:18.239279 18854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:36437 every 8 connection(s)
I20260928 13:09:18.239807 18787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:09:18.240993 18855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:09:18.241592 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 18787
I20260928 13:09:18.241706 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:09:18.241901 8546 external_mini_cluster.cc:1428] Setting key add4847a4e8327d9d6edf2debe42670c
I20260928 13:09:18.244200 18855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212: Bootstrap starting.
I20260928 13:09:18.244831 18855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212: Neither blocks nor log segments found. Creating new log.
I20260928 13:09:18.245220 18855 log.cc:824] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212: Log is configured to *not* fsync() on all Append() calls
I20260928 13:09:18.245959 18855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212: No bootstrap required, opened a new log
I20260928 13:09:18.248011 18855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2f579ffbff41ac990817dd63144212" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36437 } }
I20260928 13:09:18.248203 18855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:09:18.248256 18855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2f579ffbff41ac990817dd63144212, State: Initialized, Role: FOLLOWER
I20260928 13:09:18.248428 18855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2f579ffbff41ac990817dd63144212" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36437 } }
I20260928 13:09:18.248544 18855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:09:18.248577 18855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:09:18.248620 18855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 13:09:18 dist-test-slave-p1lc krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:09:18.249413 18855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2f579ffbff41ac990817dd63144212" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36437 } }
I20260928 13:09:18.249504 18855 leader_election.cc:302] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e2f579ffbff41ac990817dd63144212; no voters:
I20260928 13:09:18.249769 18855 leader_election.cc:288] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:09:18.249802 18860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:09:18.249914 18860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2f579ffbff41ac990817dd63144212, State: Running, Role: LEADER
I20260928 13:09:18.250000 18860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2f579ffbff41ac990817dd63144212" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36437 } }
I20260928 13:09:18.250058 18855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:09:18.250663 18861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e2f579ffbff41ac990817dd63144212. Latest consensus state: current_term: 1 leader_uuid: "1e2f579ffbff41ac990817dd63144212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2f579ffbff41ac990817dd63144212" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36437 } } }
I20260928 13:09:18.250837 18861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [sys.catalog]: This master's current role is: LEADER
I20260928 13:09:18.250684 18862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e2f579ffbff41ac990817dd63144212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2f579ffbff41ac990817dd63144212" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36437 } } }
I20260928 13:09:18.251080 18862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212 [sys.catalog]: This master's current role is: LEADER
W20260928 13:09:18.251950 18875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:09:18.252051 18875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:09:18.252156 18869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:09:18.252400 18869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:09:18.253896 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.242544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.8.88.190:36437)
0928 13:09:18.242730 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:18.242739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:18.242766 (+ 27us) server_negotiation.cc:407] Connection header received
0928 13:09:18.243193 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:18.243200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:18.243247 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:18.243367 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:18.244169 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.245405 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.246221 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.246657 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.246779 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.249710 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:18.249715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:18.249715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:09:18.249729 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:18.252014 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:09:18.252465 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:09:18.252469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:09:18.252469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:09:18.252583 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:09:18.252839 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:09:18.252840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:09:18.252840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:09:18.253199 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:18.253357 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:18.253544 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 13:09:18.253695 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260928 13:09:18.254668 18869 catalog_manager.cc:1347] Generated new cluster ID: 2cec1af6fcfc4746bd885f37e97badc5
I20260928 13:09:18.254766 18869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:09:18.301188 18869 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:09:18.302006 18869 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:09:18.314237 18869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212: Generated new TSK 0
I20260928 13:09:18.314472 18869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:09:18.353217 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36437
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:09:18.437047 18881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:18.437265 18881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:18.437295 18881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:18.438593 18881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:18.438650 18881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:18.438680 18881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:09:18.440238 18881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:18.440526 18881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:18.440788 18881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:18.446321 18889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:18.446310 18886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:18.446355 18881 server_base.cc:1034] running on GCE node
W20260928 13:09:18.446314 18887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:18.446784 18881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:18.447129 18881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:18.448311 18881 hybrid_clock.cc:648] HybridClock initialized: now 1790600958448276 us; error 69 us; skew 500 ppm
Sep 28 13:09:18 dist-test-slave-p1lc krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600958, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:09:18.451296 18881 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:09:18.451859 18881 webserver.cc:460] Webserver started at http://127.8.88.129:34073/ using document root <none> and password file <none>
I20260928 13:09:18.452105 18881 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:18.452174 18881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:18.452313 18881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:09:18.453614 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "7e2f02a4e59e4bedb7014295e1f39c8c"
format_stamp: "Formatted at 2026-09-28 13:09:18 on dist-test-slave-p1lc"
server_key: "fbe39bd0c8c3567d9e72c8a29785b459"
server_key_iv: "60b85f0aa5d5fc519f7c2e00c5a4f7d5"
server_key_version: "encryptionkey@0"
I20260928 13:09:18.453992 18881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "7e2f02a4e59e4bedb7014295e1f39c8c"
format_stamp: "Formatted at 2026-09-28 13:09:18 on dist-test-slave-p1lc"
server_key: "fbe39bd0c8c3567d9e72c8a29785b459"
server_key_iv: "60b85f0aa5d5fc519f7c2e00c5a4f7d5"
server_key_version: "encryptionkey@0"
I20260928 13:09:18.455355 18881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 13:09:18.456212 18896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:18.456401 18881 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:09:18.456552 18881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "7e2f02a4e59e4bedb7014295e1f39c8c"
format_stamp: "Formatted at 2026-09-28 13:09:18 on dist-test-slave-p1lc"
server_key: "fbe39bd0c8c3567d9e72c8a29785b459"
server_key_iv: "60b85f0aa5d5fc519f7c2e00c5a4f7d5"
server_key_version: "encryptionkey@0"
I20260928 13:09:18.456632 18881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:18.468344 18881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:18.470667 18881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:18.470844 18881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:18.471110 18881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:09:18.471431 18881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:09:18.471490 18881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:18.471534 18881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:09:18.471560 18881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:18.478282 18881 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:42533
I20260928 13:09:18.478322 19009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:42533 every 8 connection(s)
I20260928 13:09:18.478670 18881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
Sep 28 13:09:18 dist-test-slave-p1lc krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600958, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:09:18.488613 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 18881
I20260928 13:09:18.488727 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:09:18.488974 8546 external_mini_cluster.cc:1428] Setting key d1c9b1fae2e97c57b458e288bdaf9e73
I20260928 13:09:18.489181 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.479565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:56483 (local address 127.8.88.190:36437)
0928 13:09:18.479635 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:18.479637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:18.479881 (+ 244us) server_negotiation.cc:407] Connection header received
0928 13:09:18.480421 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:18.480422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:18.480433 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:18.480487 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:18.481443 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.481984 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.483065 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.483270 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.483327 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.485985 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:18.485986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:18.485986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:09:18.485994 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:18.487407 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:09:18.487861 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:09:18.487863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:09:18.487863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:09:18.487922 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:09:18.488177 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:09:18.488178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:09:18.488179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:09:18.488304 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:18.488363 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:18.488931 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 13:09:18.489058 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 13:09:18.489346 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.479552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:36437 (local address 127.8.88.129:56483)
0928 13:09:18.479758 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:18.479797 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:18.480327 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:18.480547 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:18.480550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:18.481155 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:18.481352 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.481357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.482106 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.482108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.482956 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.482959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.483372 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.483373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.483503 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.484199 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:18.484208 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:18.485881 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:09:18.487515 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:09:18.487522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:09:18.487524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:09:18.487770 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:09:18.487993 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:09:18.487993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:09:18.487993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:09:18.488104 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:09:18.488408 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:09:18.488417 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:09:18.488619 (+ 202us) client_negotiation.cc:769] Sending connection context
0928 13:09:18.488763 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 13:09:18.489083 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
I20260928 13:09:18.489893 19010 heartbeater.cc:347] Connected to a master server at 127.8.88.190:36437
I20260928 13:09:18.489997 19010 heartbeater.cc:464] Registering TS with master...
I20260928 13:09:18.490182 19010 heartbeater.cc:511] Master 127.8.88.190:36437 requested a full tablet report, sending...
I20260928 13:09:18.490777 18819 ts_manager.cc:194] Registered new tserver with Master: 7e2f02a4e59e4bedb7014295e1f39c8c (127.8.88.129:42533)
I20260928 13:09:18.491590 18819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:56483
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:09:18.517239 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36437
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:09:18.603179 19015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:18.603495 19015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:18.603541 19015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:18.604912 19015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:18.605000 19015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:18.605049 19015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:09:18.606855 19015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:18.607167 19015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:18.607424 19015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:18.613313 19023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:18.613313 19020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:18.613334 19021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:18.613365 19015 server_base.cc:1034] running on GCE node
I20260928 13:09:18.613760 19015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:18.613991 19015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:18.615358 19015 hybrid_clock.cc:648] HybridClock initialized: now 1790600958615335 us; error 30 us; skew 500 ppm
Sep 28 13:09:18 dist-test-slave-p1lc krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600958, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:09:18.618307 19015 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:09:18.618832 19015 webserver.cc:460] Webserver started at http://127.8.88.130:45233/ using document root <none> and password file <none>
I20260928 13:09:18.619091 19015 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:18.619165 19015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:18.619335 19015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:09:18.620487 19015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "2ef76f0646c14429b7ff2ed5d8c814bf"
format_stamp: "Formatted at 2026-09-28 13:09:18 on dist-test-slave-p1lc"
server_key: "686b2586195ee8068aa89ad9996a7102"
server_key_iv: "84ce59cd62a39fd7ef8dd20889b93363"
server_key_version: "encryptionkey@0"
I20260928 13:09:18.620853 19015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "2ef76f0646c14429b7ff2ed5d8c814bf"
format_stamp: "Formatted at 2026-09-28 13:09:18 on dist-test-slave-p1lc"
server_key: "686b2586195ee8068aa89ad9996a7102"
server_key_iv: "84ce59cd62a39fd7ef8dd20889b93363"
server_key_version: "encryptionkey@0"
I20260928 13:09:18.622429 19015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:09:18.623379 19030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:18.623690 19015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:09:18.623821 19015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "2ef76f0646c14429b7ff2ed5d8c814bf"
format_stamp: "Formatted at 2026-09-28 13:09:18 on dist-test-slave-p1lc"
server_key: "686b2586195ee8068aa89ad9996a7102"
server_key_iv: "84ce59cd62a39fd7ef8dd20889b93363"
server_key_version: "encryptionkey@0"
I20260928 13:09:18.623906 19015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:18.641114 19015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:18.643610 19015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:18.643766 19015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:18.643980 19015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:09:18.644251 19015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:09:18.644281 19015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:18.644305 19015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:09:18.644318 19015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:18.650461 19015 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:40325
I20260928 13:09:18.650592 19143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:40325 every 8 connection(s)
I20260928 13:09:18.650942 19015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:09:18.652760 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 19015
I20260928 13:09:18.652858 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:09:18.653069 8546 external_mini_cluster.cc:1428] Setting key 42410fac3374c22ca082b0f3b3405b28
Sep 28 13:09:18 dist-test-slave-p1lc krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600958, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
I20260928 13:09:18.661984 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.651771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:40831 (local address 127.8.88.190:36437)
0928 13:09:18.651832 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:18.651833 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:18.652127 (+ 294us) server_negotiation.cc:407] Connection header received
0928 13:09:18.652687 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:18.652688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:18.652701 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:18.652750 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:18.653951 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.654487 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.656126 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.656320 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.656384 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.658923 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:18.658924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:18.658925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:09:18.658935 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:18.660456 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:09:18.660964 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:09:18.660966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:09:18.660967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:09:18.661029 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:09:18.661292 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:09:18.661293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:09:18.661293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:09:18.661415 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:18.661488 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:18.661787 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 13:09:18.661886 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 13:09:18.662182 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.651738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:36437 (local address 127.8.88.130:40831)
0928 13:09:18.651927 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:18.652037 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:18.652567 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:18.652823 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:18.652827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:18.653604 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:18.653845 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.653849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.654589 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.654590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.655987 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.655990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.656414 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.656415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.656572 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.657187 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:18.657198 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:18.658817 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:09:18.660547 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:09:18.660556 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:09:18.660559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:09:18.660845 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:09:18.661103 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:09:18.661104 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:09:18.661105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:09:18.661216 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:09:18.661488 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:09:18.661496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:09:18.661646 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 13:09:18.661751 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 13:09:18.661935 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":97,"threads_started":1}
I20260928 13:09:18.662796 19144 heartbeater.cc:347] Connected to a master server at 127.8.88.190:36437
I20260928 13:09:18.662899 19144 heartbeater.cc:464] Registering TS with master...
I20260928 13:09:18.663107 19144 heartbeater.cc:511] Master 127.8.88.190:36437 requested a full tablet report, sending...
I20260928 13:09:18.663503 18819 ts_manager.cc:194] Registered new tserver with Master: 2ef76f0646c14429b7ff2ed5d8c814bf (127.8.88.130:40325)
I20260928 13:09:18.663879 18819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:40831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:09:18.685514 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36437
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:41269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:09:18.771770 19149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:09:18.772012 19149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:09:18.772039 19149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:09:18.773381 19149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:09:18.773429 19149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:09:18.773458 19149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:09:18.774967 19149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:09:18.775229 19149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:09:18.775482 19149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:09:18.780956 19155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:18.780961 19154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:09:18.780956 19157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:09:18.781253 19149 server_base.cc:1034] running on GCE node
I20260928 13:09:18.781489 19149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:09:18.781726 19149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:09:18.782948 19149 hybrid_clock.cc:648] HybridClock initialized: now 1790600958782908 us; error 98 us; skew 500 ppm
Sep 28 13:09:18 dist-test-slave-p1lc krb5kdc[18774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600958, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:09:18.785823 19149 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:09:18.786260 19149 webserver.cc:460] Webserver started at http://127.8.88.131:36349/ using document root <none> and password file <none>
I20260928 13:09:18.786527 19149 fs_manager.cc:359] Metadata directory not provided
I20260928 13:09:18.786603 19149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:09:18.786762 19149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:09:18.787848 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "37fef146b0c64b74940dc625e28ec91f"
format_stamp: "Formatted at 2026-09-28 13:09:18 on dist-test-slave-p1lc"
server_key: "1e48be392023202dd7ba07e3f4d11fc3"
server_key_iv: "bc7e100a47bfcfb225079164fbd26a31"
server_key_version: "encryptionkey@0"
I20260928 13:09:18.788226 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "37fef146b0c64b74940dc625e28ec91f"
format_stamp: "Formatted at 2026-09-28 13:09:18 on dist-test-slave-p1lc"
server_key: "1e48be392023202dd7ba07e3f4d11fc3"
server_key_iv: "bc7e100a47bfcfb225079164fbd26a31"
server_key_version: "encryptionkey@0"
I20260928 13:09:18.789757 19149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:09:18.790761 19164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:18.791082 19149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:09:18.791273 19149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "37fef146b0c64b74940dc625e28ec91f"
format_stamp: "Formatted at 2026-09-28 13:09:18 on dist-test-slave-p1lc"
server_key: "1e48be392023202dd7ba07e3f4d11fc3"
server_key_iv: "bc7e100a47bfcfb225079164fbd26a31"
server_key_version: "encryptionkey@0"
I20260928 13:09:18.791406 19149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:09:18.821594 19149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:09:18.823943 19149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:09:18.824144 19149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:09:18.824406 19149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:09:18.824733 19149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:09:18.824795 19149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:18.824838 19149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:09:18.824894 19149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:09:18.831467 19149 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:35411
I20260928 13:09:18.831545 19277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:35411 every 8 connection(s)
I20260928 13:09:18.831903 19149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
Sep 28 13:09:18 dist-test-slave-p1lc krb5kdc[18774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600958, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:09:18.841243 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 19149
I20260928 13:09:18.841356 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:09:18.841555 8546 external_mini_cluster.cc:1428] Setting key 346294130a090a07fd902dc9defb35e9
I20260928 13:09:18.844197 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.832729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:55047 (local address 127.8.88.190:36437)
0928 13:09:18.832779 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:18.832781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:18.833000 (+ 219us) server_negotiation.cc:407] Connection header received
0928 13:09:18.833574 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:18.833576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:18.833588 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:18.833665 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:18.834962 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.835544 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.837158 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.837361 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.837433 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.840506 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:18.840508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:18.840508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:09:18.840518 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:18.842619 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:09:18.843082 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:09:18.843083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:09:18.843085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:09:18.843157 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:09:18.843440 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:09:18.843442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:09:18.843442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:09:18.843604 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:18.843687 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:18.844002 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 13:09:18.844100 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:09:18.844422 19280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.832677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:36437 (local address 127.8.88.131:55047)
0928 13:09:18.832841 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:18.832918 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:18.833407 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:18.833786 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:18.833791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:18.834580 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:09:18.834810 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.834816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.835764 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.835765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.836831 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.836834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.837459 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.837460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.837582 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.838449 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:09:18.838458 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:09:18.840359 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:09:18.842696 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:09:18.842706 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:09:18.842709 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:09:18.842970 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:09:18.843233 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:09:18.843234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:09:18.843234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:09:18.843361 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:09:18.843676 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:09:18.843683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:09:18.843847 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 13:09:18.843968 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 13:09:18.844164 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260928 13:09:18.845013 19278 heartbeater.cc:347] Connected to a master server at 127.8.88.190:36437
I20260928 13:09:18.845113 19278 heartbeater.cc:464] Registering TS with master...
I20260928 13:09:18.845337 19278 heartbeater.cc:511] Master 127.8.88.190:36437 requested a full tablet report, sending...
I20260928 13:09:18.845825 18819 ts_manager.cc:194] Registered new tserver with Master: 37fef146b0c64b74940dc625e28ec91f (127.8.88.131:35411)
I20260928 13:09:18.846294 18819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:55047
I20260928 13:09:18.854162 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:09:18.863744 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.855776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.8.88.190:36437)
0928 13:09:18.855875 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:18.855877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:18.855994 (+ 117us) server_negotiation.cc:407] Connection header received
0928 13:09:18.856085 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:18.856086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:18.856097 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:18.856177 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:09:18.856805 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.857493 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.858332 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.858557 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.858653 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.860061 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:09:18.860062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:09:18.860063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:09:18.860072 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:09:18.862089 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:09:18.862657 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:09:18.862659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:09:18.862659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:09:18.862730 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:09:18.862986 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:09:18.862987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:09:18.862988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:09:18.863227 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:09:18.863344 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:18.863500 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 13:09:18.863602 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 13:09:18.866503 18819 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:09:18.869616 18819 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:09:18.877028 19289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.872686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:35411 (local address 127.0.0.1:56206)
0928 13:09:18.872923 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:18.872934 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:18.873020 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:18.873359 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:18.873360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:18.873361 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:09:18.873399 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.873400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.874365 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.874367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.875757 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.875759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.876655 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.876657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.876743 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.876757 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 13:09:18.876861 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 13:09:18.876946 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":67,"threads_started":1}
I20260928 13:09:18.877039 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.872966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.8.88.131:35411)
0928 13:09:18.873188 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:18.873193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:18.873204 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:09:18.873250 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:18.873252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:18.873269 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:18.873347 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:09:18.873482 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.874234 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.875848 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.876552 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.876651 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.876718 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:18.876900 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 13:09:18.876944 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
I20260928 13:09:18.877467 19290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.872462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:42533 (local address 127.0.0.1:37390)
0928 13:09:18.872711 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:18.872761 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:18.872850 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:18.873197 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:18.873201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:18.873203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:09:18.873286 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.873289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.875109 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.875110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.876221 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.876224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.877228 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.877229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.877287 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.877290 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:09:18.877345 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 13:09:18.877380 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":70,"threads_started":1}
I20260928 13:09:18.877367 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.872510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:40325 (local address 127.0.0.1:49438)
0928 13:09:18.872903 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:18.872913 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:18.873020 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:18.873691 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:18.873692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:18.873695 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:09:18.873755 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.873758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.874932 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.874933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.876046 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.876048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.877092 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.877093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.877159 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.877163 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:09:18.877212 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 13:09:18.877265 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":7,"thread_start_us":133,"threads_started":1}
I20260928 13:09:18.877604 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.872738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.8.88.130:40325)
0928 13:09:18.873500 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:18.873510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:18.873519 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:09:18.873551 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:18.873552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:18.873565 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:18.873690 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:09:18.873822 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.874808 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.876141 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.876918 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.876996 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.877052 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:18.877456 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 13:09:18.877516 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":97,"threads_started":1}
I20260928 13:09:18.878248 19078 tablet_service.cc:1467] Processing CreateTablet for tablet 97f8799529b5422cb9f92bd103bc1366 (DEFAULT_TABLE table=test-table [id=a9e7f2517672469999786551742f0446]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:09:18.878250 19212 tablet_service.cc:1467] Processing CreateTablet for tablet 97f8799529b5422cb9f92bd103bc1366 (DEFAULT_TABLE table=test-table [id=a9e7f2517672469999786551742f0446]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:09:18.878590 19212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f8799529b5422cb9f92bd103bc1366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:09:18.878605 19078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f8799529b5422cb9f92bd103bc1366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:09:18.881021 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.872591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.8.88.129:42533)
0928 13:09:18.872777 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:18.872782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:18.872824 (+ 42us) server_negotiation.cc:407] Connection header received
0928 13:09:18.873044 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:18.873048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:18.873073 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:18.873171 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:09:18.874335 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.875001 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.876314 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.877084 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.877145 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.877197 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:18.877378 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 13:09:18.877426 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
I20260928 13:09:18.881886 19296 tablet_bootstrap.cc:492] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f: Bootstrap starting.
I20260928 13:09:18.881915 19295 tablet_bootstrap.cc:492] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf: Bootstrap starting.
I20260928 13:09:18.881800 18944 tablet_service.cc:1467] Processing CreateTablet for tablet 97f8799529b5422cb9f92bd103bc1366 (DEFAULT_TABLE table=test-table [id=a9e7f2517672469999786551742f0446]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:09:18.882216 18944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f8799529b5422cb9f92bd103bc1366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:09:18.882597 19296 tablet_bootstrap.cc:654] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f: Neither blocks nor log segments found. Creating new log.
I20260928 13:09:18.882604 19295 tablet_bootstrap.cc:654] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf: Neither blocks nor log segments found. Creating new log.
I20260928 13:09:18.883085 19296 log.cc:824] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f: Log is configured to *not* fsync() on all Append() calls
I20260928 13:09:18.883144 19295 log.cc:824] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf: Log is configured to *not* fsync() on all Append() calls
I20260928 13:09:18.883870 19296 tablet_bootstrap.cc:492] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f: No bootstrap required, opened a new log
I20260928 13:09:18.883905 19295 tablet_bootstrap.cc:492] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf: No bootstrap required, opened a new log
I20260928 13:09:18.883960 19296 ts_tablet_manager.cc:1397] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:09:18.883973 19295 ts_tablet_manager.cc:1397] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:09:18.885612 19296 raft_consensus.cc:348] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40325 } } peers { permanent_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42533 } } peers { permanent_uuid: "37fef146b0c64b74940dc625e28ec91f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35411 } }
I20260928 13:09:18.885787 19296 raft_consensus.cc:374] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:09:18.885830 19296 raft_consensus.cc:729] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37fef146b0c64b74940dc625e28ec91f, State: Initialized, Role: FOLLOWER
I20260928 13:09:18.886071 19299 tablet_bootstrap.cc:492] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c: Bootstrap starting.
I20260928 13:09:18.886068 19296 consensus_queue.cc:260] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40325 } } peers { permanent_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42533 } } peers { permanent_uuid: "37fef146b0c64b74940dc625e28ec91f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35411 } }
I20260928 13:09:18.886119 19295 raft_consensus.cc:348] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40325 } } peers { permanent_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42533 } } peers { permanent_uuid: "37fef146b0c64b74940dc625e28ec91f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35411 } }
I20260928 13:09:18.886314 19295 raft_consensus.cc:374] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:09:18.886363 19295 raft_consensus.cc:729] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef76f0646c14429b7ff2ed5d8c814bf, State: Initialized, Role: FOLLOWER
I20260928 13:09:18.886406 19296 ts_tablet_manager.cc:1428] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:09:18.886497 19295 consensus_queue.cc:260] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40325 } } peers { permanent_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42533 } } peers { permanent_uuid: "37fef146b0c64b74940dc625e28ec91f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35411 } }
I20260928 13:09:18.886560 19278 heartbeater.cc:503] Master 127.8.88.190:36437 was elected leader, sending a full tablet report...
I20260928 13:09:18.886847 19299 tablet_bootstrap.cc:654] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c: Neither blocks nor log segments found. Creating new log.
I20260928 13:09:18.886781 19295 ts_tablet_manager.cc:1428] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:09:18.886832 19144 heartbeater.cc:503] Master 127.8.88.190:36437 was elected leader, sending a full tablet report...
I20260928 13:09:18.887526 19299 log.cc:824] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c: Log is configured to *not* fsync() on all Append() calls
I20260928 13:09:18.888547 19299 tablet_bootstrap.cc:492] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c: No bootstrap required, opened a new log
I20260928 13:09:18.888635 19299 ts_tablet_manager.cc:1397] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 13:09:18.890271 19299 raft_consensus.cc:348] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40325 } } peers { permanent_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42533 } } peers { permanent_uuid: "37fef146b0c64b74940dc625e28ec91f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35411 } }
I20260928 13:09:18.890429 19299 raft_consensus.cc:374] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:09:18.890456 19299 raft_consensus.cc:729] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e2f02a4e59e4bedb7014295e1f39c8c, State: Initialized, Role: FOLLOWER
I20260928 13:09:18.890584 19299 consensus_queue.cc:260] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40325 } } peers { permanent_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42533 } } peers { permanent_uuid: "37fef146b0c64b74940dc625e28ec91f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35411 } }
I20260928 13:09:18.890852 19299 ts_tablet_manager.cc:1428] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:09:18.890897 19010 heartbeater.cc:503] Master 127.8.88.190:36437 was elected leader, sending a full tablet report...
W20260928 13:09:18.901551 19145 tablet.cc:2369] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:09:18.914338 19301 raft_consensus.cc:482] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:09:18.914476 19301 raft_consensus.cc:504] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40325 } } peers { permanent_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42533 } } peers { permanent_uuid: "37fef146b0c64b74940dc625e28ec91f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35411 } }
I20260928 13:09:18.914940 19301 leader_election.cc:288] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e2f02a4e59e4bedb7014295e1f39c8c (127.8.88.129:42533), 37fef146b0c64b74940dc625e28ec91f (127.8.88.131:35411)
I20260928 13:09:18.918128 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.915118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:42533 (local address 127.8.88.130:49965)
0928 13:09:18.915200 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:18.915213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:18.915266 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:18.915495 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:18.915497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:18.915499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:09:18.915583 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.915585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.916339 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.916340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.917194 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.917196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.917853 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.917854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.917938 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.917942 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:09:18.918010 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 13:09:18.918052 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 13:09:18.918128 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.915240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:49965 (local address 127.8.88.129:42533)
0928 13:09:18.915308 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:18.915311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:18.915321 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:09:18.915353 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:18.915354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:18.915368 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:18.915480 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:09:18.915668 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.916235 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.917288 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.917766 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.917857 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.917867 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:18.918016 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 13:09:18.918058 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 13:09:18.918406 19303 raft_consensus.cc:482] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:09:18.918480 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.915349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:46201 (local address 127.8.88.131:35411)
0928 13:09:18.915433 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:18.915435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:18.915591 (+ 156us) server_negotiation.cc:407] Connection header received
0928 13:09:18.915674 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:18.915675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:18.915684 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:18.915751 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:09:18.915920 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.916506 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.917615 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.918051 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.918141 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.918149 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:18.918350 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 13:09:18.918399 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 13:09:18.918509 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f8799529b5422cb9f92bd103bc1366" candidate_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" is_pre_election: true
I20260928 13:09:18.918514 19303 raft_consensus.cc:504] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40325 } } peers { permanent_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42533 } } peers { permanent_uuid: "37fef146b0c64b74940dc625e28ec91f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35411 } }
I20260928 13:09:18.918689 18964 raft_consensus.cc:2454] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef76f0646c14429b7ff2ed5d8c814bf in term 0.
I20260928 13:09:18.918718 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.915261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:35411 (local address 127.8.88.130:46201)
0928 13:09:18.915509 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:18.915519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:18.915600 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:18.915753 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:18.915754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:18.915755 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:09:18.915841 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.915843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.916605 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.916607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.917517 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.917520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.918141 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.918142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.918235 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.918241 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 13:09:18.918570 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 13:09:18.918628 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":130,"threads_started":1}
I20260928 13:09:18.919035 19033 leader_election.cc:302] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef76f0646c14429b7ff2ed5d8c814bf, 7e2f02a4e59e4bedb7014295e1f39c8c; no voters:
I20260928 13:09:18.919135 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f8799529b5422cb9f92bd103bc1366" candidate_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fef146b0c64b74940dc625e28ec91f" is_pre_election: true
I20260928 13:09:18.919198 19301 raft_consensus.cc:2781] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:09:18.919250 19301 raft_consensus.cc:482] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:09:18.919274 19301 raft_consensus.cc:3037] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:09:18.919299 19232 raft_consensus.cc:2454] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef76f0646c14429b7ff2ed5d8c814bf in term 0.
I20260928 13:09:18.919324 19303 leader_election.cc:288] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ef76f0646c14429b7ff2ed5d8c814bf (127.8.88.130:40325), 37fef146b0c64b74940dc625e28ec91f (127.8.88.131:35411)
I20260928 13:09:18.920338 19301 raft_consensus.cc:504] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40325 } } peers { permanent_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42533 } } peers { permanent_uuid: "37fef146b0c64b74940dc625e28ec91f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35411 } }
I20260928 13:09:18.920506 19301 leader_election.cc:288] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [CANDIDATE]: Term 1 election: Requested vote from peers 7e2f02a4e59e4bedb7014295e1f39c8c (127.8.88.129:42533), 37fef146b0c64b74940dc625e28ec91f (127.8.88.131:35411)
I20260928 13:09:18.920780 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f8799529b5422cb9f92bd103bc1366" candidate_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fef146b0c64b74940dc625e28ec91f"
I20260928 13:09:18.920804 18964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f8799529b5422cb9f92bd103bc1366" candidate_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c"
I20260928 13:09:18.920861 19232 raft_consensus.cc:3037] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:09:18.920923 18964 raft_consensus.cc:3037] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:09:18.921893 19232 raft_consensus.cc:2454] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef76f0646c14429b7ff2ed5d8c814bf in term 1.
I20260928 13:09:18.921906 18964 raft_consensus.cc:2454] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef76f0646c14429b7ff2ed5d8c814bf in term 1.
I20260928 13:09:18.922093 19033 leader_election.cc:302] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef76f0646c14429b7ff2ed5d8c814bf, 7e2f02a4e59e4bedb7014295e1f39c8c; no voters:
I20260928 13:09:18.922817 19301 raft_consensus.cc:2781] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:09:18.922936 19305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.919169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:40325 (local address 127.8.88.129:53363)
0928 13:09:18.919647 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:18.919660 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:18.919750 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:18.919931 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:18.919932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:18.919933 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:09:18.920019 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.920021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.920843 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.920844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.922067 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.922071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.922665 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.922666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.922730 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.922733 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:09:18.922806 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 13:09:18.922849 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":281,"thread_start_us":124,"threads_started":1}
I20260928 13:09:18.922935 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.919236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:53363 (local address 127.8.88.130:40325)
0928 13:09:18.919273 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:18.919275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:18.919721 (+ 446us) server_negotiation.cc:407] Connection header received
0928 13:09:18.919824 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:18.919825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:18.919835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:18.919865 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:09:18.920111 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.920760 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.922162 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.922566 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.922606 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.922613 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:18.922811 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 13:09:18.922848 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:09:18.923043 19301 raft_consensus.cc:686] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [term 1 LEADER]: Becoming Leader. State: Replica: 2ef76f0646c14429b7ff2ed5d8c814bf, State: Running, Role: LEADER
I20260928 13:09:18.923077 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.919169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:35411 (local address 127.8.88.129:51587)
0928 13:09:18.919219 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:18.919231 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:09:18.919266 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:09:18.919568 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:09:18.919569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:09:18.919571 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:09:18.919631 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.919632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.920663 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.920664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.922121 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:09:18.922125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.922767 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.922768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:09:18.922863 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.922867 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:09:18.922926 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 13:09:18.922985 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":30}
I20260928 13:09:18.923168 19301 consensus_queue.cc:237] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40325 } } peers { permanent_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42533 } } peers { permanent_uuid: "37fef146b0c64b74940dc625e28ec91f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35411 } }
I20260928 13:09:18.923245 19098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f8799529b5422cb9f92bd103bc1366" candidate_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" is_pre_election: true
I20260928 13:09:18.923687 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.919385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:51587 (local address 127.8.88.131:35411)
0928 13:09:18.919436 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:18.919438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:18.919445 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:09:18.919468 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:18.919469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:18.919480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:18.919527 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:09:18.919798 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.920530 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.922227 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.922678 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.922770 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.922778 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:18.923559 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 13:09:18.923614 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:09:18.924043 19232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97f8799529b5422cb9f92bd103bc1366" candidate_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37fef146b0c64b74940dc625e28ec91f" is_pre_election: true
I20260928 13:09:18.924224 19232 raft_consensus.cc:2379] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e2f02a4e59e4bedb7014295e1f39c8c in current term 1: Already voted for candidate 2ef76f0646c14429b7ff2ed5d8c814bf in this term.
I20260928 13:09:18.924269 18819 catalog_manager.cc:5495] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ef76f0646c14429b7ff2ed5d8c814bf (127.8.88.130). New cstate: current_term: 1 leader_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ef76f0646c14429b7ff2ed5d8c814bf" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 40325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37fef146b0c64b74940dc625e28ec91f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35411 } health_report { overall_health: UNKNOWN } } }
I20260928 13:09:18.924465 18899 leader_election.cc:302] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e2f02a4e59e4bedb7014295e1f39c8c; no voters: 2ef76f0646c14429b7ff2ed5d8c814bf, 37fef146b0c64b74940dc625e28ec91f
I20260928 13:09:18.924618 19303 raft_consensus.cc:2726] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 13:09:18.937928 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.934831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.8.88.130:40325)
0928 13:09:18.934905 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 13:09:18.934908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:09:18.934937 (+ 29us) server_negotiation.cc:407] Connection header received
0928 13:09:18.935033 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:09:18.935035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:09:18.935055 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:09:18.935153 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:09:18.935330 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.935995 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.936831 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:09:18.937161 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:09:18.937245 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:09:18.937520 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:09:18.937635 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:09:18.937746 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 13:09:18.937769 (+ 23us) server_negotiation.cc:299] Negotiation successful
0928 13:09:18.937793 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 13:09:18.940194 18964 raft_consensus.cc:1261] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [term 1 FOLLOWER]: Refusing update from remote peer 2ef76f0646c14429b7ff2ed5d8c814bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:09:18.940188 19232 raft_consensus.cc:1261] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f [term 1 FOLLOWER]: Refusing update from remote peer 2ef76f0646c14429b7ff2ed5d8c814bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:09:18.940546 19306 consensus_queue.cc:1035] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e2f02a4e59e4bedb7014295e1f39c8c" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 42533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:09:18.940681 19301 consensus_queue.cc:1035] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "37fef146b0c64b74940dc625e28ec91f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:09:18.943641 19311 mvcc.cc:205] Tried to move back new op lower bound from 7334301527815233536 to 7334301527750225920. Current Snapshot: MvccSnapshot[applied={T|T < 7334301527815233536}]
I20260928 13:09:18.944520 19310 mvcc.cc:205] Tried to move back new op lower bound from 7334301527815233536 to 7334301527750225920. Current Snapshot: MvccSnapshot[applied={T|T < 7334301527815233536}]
I20260928 13:09:18.946295 18818 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:09:18.946439 18818 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:09:18.947266 18818 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e2f579ffbff41ac990817dd63144212: Sending DeleteTablet for 3 replicas of tablet 97f8799529b5422cb9f92bd103bc1366
I20260928 13:09:18.947646 18944 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f8799529b5422cb9f92bd103bc1366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:09:18 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:37390
I20260928 13:09:18.947685 19212 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f8799529b5422cb9f92bd103bc1366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:09:18 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:56206
I20260928 13:09:18.947675 19078 tablet_service.cc:1514] Processing DeleteTablet for tablet 97f8799529b5422cb9f92bd103bc1366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:09:18 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:49438
I20260928 13:09:18.947964 19319 tablet_replica.cc:330] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f: stopping tablet replica
I20260928 13:09:18.948083 19319 raft_consensus.cc:2229] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:09:18.948206 19319 raft_consensus.cc:2258] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:09:18.948295 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 18881
I20260928 13:09:18.948390 19318 tablet_replica.cc:330] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c: stopping tablet replica
I20260928 13:09:18.948493 19318 raft_consensus.cc:2229] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:09:18.948606 19318 raft_consensus.cc:2258] T 97f8799529b5422cb9f92bd103bc1366 P 7e2f02a4e59e4bedb7014295e1f39c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:09:18.949570 19320 tablet_replica.cc:330] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf: stopping tablet replica
I20260928 13:09:18.949590 19319 ts_tablet_manager.cc:1905] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:09:18.949719 19320 raft_consensus.cc:2229] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:09:18.949833 19320 raft_consensus.cc:2258] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:09:18.950335 19320 ts_tablet_manager.cc:1905] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:09:18.951973 19319 ts_tablet_manager.cc:1918] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:09:18.952116 19319 log.cc:1196] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/97f8799529b5422cb9f92bd103bc1366
I20260928 13:09:18.952484 19319 ts_tablet_manager.cc:1939] T 97f8799529b5422cb9f92bd103bc1366 P 37fef146b0c64b74940dc625e28ec91f: Deleting consensus metadata
I20260928 13:09:18.952492 19320 ts_tablet_manager.cc:1918] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:09:18.952574 19320 log.cc:1196] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/97f8799529b5422cb9f92bd103bc1366
I20260928 13:09:18.952857 19320 ts_tablet_manager.cc:1939] T 97f8799529b5422cb9f92bd103bc1366 P 2ef76f0646c14429b7ff2ed5d8c814bf: Deleting consensus metadata
I20260928 13:09:18.953236 18805 catalog_manager.cc:4841] TS 37fef146b0c64b74940dc625e28ec91f (127.8.88.131:35411): tablet 97f8799529b5422cb9f92bd103bc1366 (table test-table [id=a9e7f2517672469999786551742f0446]) successfully deleted
I20260928 13:09:18.953415 18804 catalog_manager.cc:4841] TS 2ef76f0646c14429b7ff2ed5d8c814bf (127.8.88.130:40325): tablet 97f8799529b5422cb9f92bd103bc1366 (table test-table [id=a9e7f2517672469999786551742f0446]) successfully deleted
W20260928 13:09:18.955518 18805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:42533 (error 108)
I20260928 13:09:18.955888 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 19015
I20260928 13:09:18.958276 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:09:18.958107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:42533 (local address 127.0.0.1:37394)
0928 13:09:18.958184 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 13:09:18.958233 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:42533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 13:09:18.958453 18805 catalog_manager.cc:4584] TS 7e2f02a4e59e4bedb7014295e1f39c8c (127.8.88.129:42533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97f8799529b5422cb9f92bd103bc1366: Network error: Client connection negotiation failed: client connection to 127.8.88.129:42533: connect: Connection refused (error 111)
I20260928 13:09:18.961830 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 19149
I20260928 13:09:18.966698 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 18787
2026-09-28T13:09:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2084 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:09:19 dist-test-slave-p1lc krb5kdc[19324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:09:19 dist-test-slave-p1lc krb5kdc[19324](info): set up 2 sockets
Sep 28 13:09:19 dist-test-slave-p1lc krb5kdc[19324](info): commencing operation
krb5kdc: starting...
W20260928 13:09:20.025796 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:09:20 dist-test-slave-p1lc krb5kdc[19324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:09:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.88.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.88.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/rangeradmin_127.8.88.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/rangeradmin_127.8.88.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.88.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.88.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/rangerlookup_127.8.88.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/rangerlookup_127.8.88.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.88.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.88.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/HTTP_127.8.88.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/HTTP_127.8.88.148@KRBTEST.COM.keytab.
I20260928 13:09:20.204382 8546 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcZlb_Q/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/postgres -l logfile start
2026-09-28 13:09:23.149 UTC [19354] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 13:09:23.150 UTC [19354] LOG: listening on IPv4 address "127.8.88.148", port 52443
2026-09-28 13:09:23.155 UTC [19354] LOG: listening on Unix socket "/tmp/.s.PGSQL.52443"
2026-09-28 13:09:23.177 UTC [19357] LOG: database system was shut down at 2026-09-28 13:09:21 UTC
2026-09-28 13:09:23.181 UTC [19354] LOG: database system is ready to accept connections
I20260928 13:09:24.141031 8546 mini_postgres.cc:96] Postgres bound to 52443
2026-09-28 13:09:24.147 UTC [19365] FATAL: database "slave" does not exist
127.8.88.148:52443 - accepting connections
I20260928 13:09:24.147855 8546 mini_ranger.cc:161] Starting Ranger...
I20260928 13:09:24.163552 8546 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 13:09:24.783665 8546 mini_ranger.cc:87] Created ranger Postgres database
I20260928 13:09:24.783738 8546 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-admin
2026-09-28 13:09:25,083 [I] DB FLAVOR :POSTGRES
2026-09-28 13:09:25,084 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 13:09:25,084 [I] Checking connection..
2026-09-28 13:09:25,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 13:09:25,364 [I] Checking connection passed.
2026-09-28 13:09:25,364 [I] --------- Verifying version history table ---------
2026-09-28 13:09:25,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 13:09:25,637 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 13:09:25,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 13:09:25,904 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 13:09:25,904 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 13:09:25,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 13:09:26.176 UTC [19429] WARNING: there is no transaction in progress
2026-09-28 13:09:26,184 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 13:09:26,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 13:09:26,437 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 13:09:26,437 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 13:09:26,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 13:09:26,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 13:09:26,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p1lc.c.gcp-upstream.internal','N') ;"
2026-09-28 13:09:27,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 13:09:27,488 [I] Table x_portal_user does not exist in database ranger
2026-09-28 13:09:27,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 13:09:27,743 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 13:09:27,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 13:09:28,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 13:09:28,271 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 13:09:28,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 13:09:29.397 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.417 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.438 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.457 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.477 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.499 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.570 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.579 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.597 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.609 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.622 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.639 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.651 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.659 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.668 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:29.678 UTC [19537] WARNING: there is no transaction in progress
2026-09-28 13:09:30,527 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 13:09:30,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 13:09:30,798 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 13:09:30,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 13:09:31,082 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 13:09:31,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 13:09:31,339 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 13:09:31,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 13:09:31,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 13:09:31,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p1lc.c.gcp-upstream.internal';"
2026-09-28 13:09:32,125 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 13:09:32,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 13:09:32,388 [I] DB_PATCHES have already been applied
I20260928 13:09:32.394347 8546 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcZlb_Q/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 13:09:32.394424 8546 mini_ranger.cc:193] Using host: 127.8.88.148
I20260928 13:09:32.398165 8546 mini_ranger.cc:227] Ranger admin URL: http://127.8.88.148:55687
Sep 28, 2026 1:09:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 1:09:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 1:09:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 1:09:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 1:09:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 1:09:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 1:09:33 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 13:09:33.566100 8546 mini_ranger.cc:160] Time spent starting Ranger: real 9.418s user 0.000s sys 0.010s
Sep 28, 2026 1:09:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 1:09:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 1:09:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 1:09:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 1:09:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 1:09:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 1:09:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 1:09:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 1:09:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 1:09:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 1:09:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 1:09:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 1:09:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 1:09:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 1:09:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 1:09:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 1:09:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 1:09:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 13:09:56.634253 8546 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.88.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.88.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/rangerkms_127.8.88.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/rangerkms_127.8.88.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.88.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.88.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.88.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/HTTP_127.8.88.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.88.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/HTTP_127.8.88.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 13:09:56 dist-test-slave-p1lc krb5kdc[19324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600996, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 13:09:56.723678 8546 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 13:09:56.734246 8546 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 13:09:57.803834 8546 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 13:09:57.803930 8546 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-kms
I20260928 13:09:57.803953 8546 mini_ranger_kms.cc:227] Using postgres at 127.8.88.148:52443
2026-09-28 13:09:57,995 [I] DB FLAVOR :POSTGRES
2026-09-28 13:09:57,995 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 13:09:57,995 [I] Checking connection
2026-09-28 13:09:57,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 13:09:58,259 [I] connection success
2026-09-28 13:09:58,260 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 13:09:58,260 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 13:09:58,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 13:09:58,504 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 13:09:58,505 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 13:09:58,505 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 13:09:58,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:52443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 13:09:58,783 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 13:09:59.363862 8546 mini_ranger_kms.cc:310] Created kms service
I20260928 13:09:59.600361 8546 mini_ranger_kms.cc:326] Created kudu user
I20260928 13:09:59.655850 8546 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 13:09:59 dist-test-slave-p1lc krb5kdc[19324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790600996, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:00.107740 8546 mini_ranger_kms.cc:384] Added ranger policy
I20260928 13:10:00.108042 8546 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 13:10:00.108078 8546 mini_ranger_kms.cc:240] Using host: 127.8.88.148
I20260928 13:10:00.112332 8546 mini_ranger_kms.cc:276] Ranger KMS PID: 19718
I20260928 13:10:00.112479 8546 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.88.148:32979
Sep 28, 2026 1:10:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 1:10:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 1:10:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 1:10:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 1:10:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 1:10:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 13:10:02.377964 8546 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.654s user 0.001s sys 0.170s
I20260928 13:10:02.378091 8546 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 13:10:02.378182 8546 mini_ranger_kms.cc:401] 127.8.88.148:32979/kms/v1/keys
Sep 28 13:10:02 dist-test-slave-p1lc krb5kdc[19324](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790601002, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.88.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 13:10:02 dist-test-slave-p1lc krb5kdc[19324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601002, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.88.148@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
Sep 28 13:10:03 dist-test-slave-p1lc krb5kdc[19324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601002, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.88.148@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
Sep 28 13:10:05 dist-test-slave-p1lc krb5kdc[19324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:10:05.612315 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:36031
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:36031
--ranger_config_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.88.148:32979/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:10:05.729012 19768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:10:05.729245 19768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:10:05.729270 19768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:10:05.730568 19768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:10:05.730623 19768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:10:05.730638 19768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:10:05.730654 19768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:10:05.730665 19768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:10:05.732374 19768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.88.148:32979/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:36031
--ranger_config_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:36031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:10:05.732653 19768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:10:05.732981 19768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:10:05.738803 19773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:05.738883 19768 server_base.cc:1034] running on GCE node
W20260928 13:10:05.738816 19774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:05.738891 19776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:05.739414 19768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:10:05.739706 19768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:10:05.740911 19768 hybrid_clock.cc:648] HybridClock initialized: now 1790601005740890 us; error 28 us; skew 500 ppm
Sep 28 13:10:05 dist-test-slave-p1lc krb5kdc[19324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601005, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:10:05.751749 19768 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:10:05.752308 19768 webserver.cc:460] Webserver started at http://127.8.88.190:42677/ using document root <none> and password file <none>
I20260928 13:10:05.752568 19768 fs_manager.cc:359] Metadata directory not provided
I20260928 13:10:05.752612 19768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:10:05.752746 19768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 13:10:05 dist-test-slave-p1lc krb5kdc[19324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601005, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:05.816922 19768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "0b369f00fa684b2e80dc508c6ca5ffc9"
format_stamp: "Formatted at 2026-09-28 13:10:05 on dist-test-slave-p1lc"
server_key: "89c29efb17cc87c474c6b6b0607e606d"
server_key_iv: "817dda48977cfdb213b71bd34efcb33e"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:05.817483 19768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "0b369f00fa684b2e80dc508c6ca5ffc9"
format_stamp: "Formatted at 2026-09-28 13:10:05 on dist-test-slave-p1lc"
server_key: "89c29efb17cc87c474c6b6b0607e606d"
server_key_iv: "817dda48977cfdb213b71bd34efcb33e"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:05.818965 19768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:10:05.835955 19783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:05.836221 19768 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:10:05.836323 19768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "0b369f00fa684b2e80dc508c6ca5ffc9"
format_stamp: "Formatted at 2026-09-28 13:10:05 on dist-test-slave-p1lc"
server_key: "89c29efb17cc87c474c6b6b0607e606d"
server_key_iv: "817dda48977cfdb213b71bd34efcb33e"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:05.836418 19768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:10:05.872823 19768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:10:05.875205 19768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:10:05.875337 19768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:10:05.881541 19768 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:36031
I20260928 13:10:05.881562 19835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:36031 every 8 connection(s)
I20260928 13:10:05.882030 19768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:10:05.882879 19836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:10:05.886300 19836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9: Bootstrap starting.
I20260928 13:10:05.887059 19836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9: Neither blocks nor log segments found. Creating new log.
I20260928 13:10:05.887416 19836 log.cc:824] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9: Log is configured to *not* fsync() on all Append() calls
I20260928 13:10:05.888334 19836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9: No bootstrap required, opened a new log
I20260928 13:10:05.891162 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 19768
I20260928 13:10:05.891290 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:10:05.891482 8546 external_mini_cluster.cc:1428] Setting key a3e8b4d13de6adee5eec9c9a4a544a47
I20260928 13:10:05.891444 19836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b369f00fa684b2e80dc508c6ca5ffc9" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36031 } }
I20260928 13:10:05.891665 19836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:10:05.891762 19836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b369f00fa684b2e80dc508c6ca5ffc9, State: Initialized, Role: FOLLOWER
I20260928 13:10:05.891973 19836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b369f00fa684b2e80dc508c6ca5ffc9" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36031 } }
I20260928 13:10:05.892179 19836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:10:05.892270 19836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:10:05.892330 19836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:10:05.893213 19836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b369f00fa684b2e80dc508c6ca5ffc9" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36031 } }
I20260928 13:10:05.893337 19836 leader_election.cc:302] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b369f00fa684b2e80dc508c6ca5ffc9; no voters:
I20260928 13:10:05.893543 19836 leader_election.cc:288] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:10:05.893607 19841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:10:05.893842 19841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [term 1 LEADER]: Becoming Leader. State: Replica: 0b369f00fa684b2e80dc508c6ca5ffc9, State: Running, Role: LEADER
I20260928 13:10:05.893904 19836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:10:05.894025 19841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b369f00fa684b2e80dc508c6ca5ffc9" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36031 } }
I20260928 13:10:05.894429 19841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b369f00fa684b2e80dc508c6ca5ffc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b369f00fa684b2e80dc508c6ca5ffc9" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36031 } } }
I20260928 13:10:05.894589 19841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [sys.catalog]: This master's current role is: LEADER
I20260928 13:10:05.894757 19841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b369f00fa684b2e80dc508c6ca5ffc9. Latest consensus state: current_term: 1 leader_uuid: "0b369f00fa684b2e80dc508c6ca5ffc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b369f00fa684b2e80dc508c6ca5ffc9" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36031 } } }
I20260928 13:10:05.894951 19841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9 [sys.catalog]: This master's current role is: LEADER
I20260928 13:10:05.895116 19850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:10:05.895430 19836 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 13:10:05.895823 19850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 13:10:05 dist-test-slave-p1lc krb5kdc[19324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:10:05.908504 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:05.892200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.8.88.190:36031)
0928 13:10:05.892412 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:05.892422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:05.892453 (+ 31us) server_negotiation.cc:407] Connection header received
0928 13:10:05.892567 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:05.892572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:05.892617 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:05.892770 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:05.893570 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:05.895187 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:05.898016 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:05.900163 (+ 2147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:05.900341 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:05.903761 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:05.903765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:05.903766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:05.903780 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:05.906018 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:05.906510 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:05.906515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:05.906515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:05.906650 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:05.906870 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:05.906871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:05.906871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:05.907229 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:05.907384 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:05.908019 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 13:10:05.908205 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":99,"threads_started":1}
I20260928 13:10:05.913445 19850 catalog_manager.cc:1347] Generated new cluster ID: 829cf5abc2de445f9be93020c2611394
I20260928 13:10:05.913556 19850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:10:05.927155 19850 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:10:05.928143 19850 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:10:05.938730 19850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9: Generated new TSK 0
I20260928 13:10:05.939116 19850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:10:06.677382 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.88.148:32979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36031
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:10:06.801111 19881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:10:06.801386 19881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:10:06.801430 19881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:10:06.802838 19881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:10:06.802906 19881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:10:06.802954 19881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:10:06.805303 19881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.88.148:32979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:10:06.805620 19881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:10:06.805936 19881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:10:06.813937 19890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:06.813934 19888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:06.814697 19881 server_base.cc:1034] running on GCE node
W20260928 13:10:06.817058 19887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:06.817384 19881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:10:06.817756 19881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:10:06.818958 19881 hybrid_clock.cc:648] HybridClock initialized: now 1790601006818933 us; error 50 us; skew 500 ppm
Sep 28 13:10:06 dist-test-slave-p1lc krb5kdc[19324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601006, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:10:06.822288 19881 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:10:06.822887 19881 webserver.cc:460] Webserver started at http://127.8.88.129:35899/ using document root <none> and password file <none>
I20260928 13:10:06.823166 19881 fs_manager.cc:359] Metadata directory not provided
I20260928 13:10:06.823241 19881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:10:06.823386 19881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 13:10:06 dist-test-slave-p1lc krb5kdc[19324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601006, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:06.837760 19881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "5228f778fcbf40008ddcb1066d6d3a99"
format_stamp: "Formatted at 2026-09-28 13:10:06 on dist-test-slave-p1lc"
server_key: "8bd4ba74bab80f5d25303ff9587d8694"
server_key_iv: "8a75992b7133542d8cd9ee1e621d6778"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:06.838203 19881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "5228f778fcbf40008ddcb1066d6d3a99"
format_stamp: "Formatted at 2026-09-28 13:10:06 on dist-test-slave-p1lc"
server_key: "8bd4ba74bab80f5d25303ff9587d8694"
server_key_iv: "8a75992b7133542d8cd9ee1e621d6778"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:06.844128 19881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.001s
I20260928 13:10:06.858829 19898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:06.859493 19881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 13:10:06.859601 19881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "5228f778fcbf40008ddcb1066d6d3a99"
format_stamp: "Formatted at 2026-09-28 13:10:06 on dist-test-slave-p1lc"
server_key: "8bd4ba74bab80f5d25303ff9587d8694"
server_key_iv: "8a75992b7133542d8cd9ee1e621d6778"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:06.859709 19881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:10:06.890751 19881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:10:06.893525 19881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:10:06.893769 19881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:10:06.894063 19881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:10:06.894495 19881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:10:06.894593 19881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:06.894706 19881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:10:06.894783 19881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:06.903236 19881 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:38349
I20260928 13:10:06.903817 19881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:10:06.905833 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 19881
I20260928 13:10:06.905936 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:10:06.906169 8546 external_mini_cluster.cc:1428] Setting key a1fe905e909225770f1a15d37257acbe
I20260928 13:10:06.905406 20011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:38349 every 8 connection(s)
Sep 28 13:10:06 dist-test-slave-p1lc krb5kdc[19324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601006, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:10:06.918967 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:06.906239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:36031 (local address 127.8.88.129:57371)
0928 13:10:06.907168 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:06.907215 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:06.907829 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:06.908218 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:06.908222 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:06.908531 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:10:06.908959 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:06.908965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:06.910289 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:06.910290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:06.911590 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:06.911593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:06.912150 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:06.912151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:06.912296 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:06.912977 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:10:06.913007 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:10:06.914750 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:10:06.917119 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:06.917129 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:06.917132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:06.917460 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:06.917755 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:06.917755 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:06.917756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:06.917901 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:06.918215 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:10:06.918223 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:10:06.918418 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 13:10:06.918548 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 13:10:06.918719 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"thread_start_us":90,"threads_started":1}
I20260928 13:10:06.919309 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:06.906641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:57371 (local address 127.8.88.190:36031)
0928 13:10:06.907106 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:06.907108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:06.907974 (+ 866us) server_negotiation.cc:407] Connection header received
0928 13:10:06.908002 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:06.908003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:06.908018 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:06.908080 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:06.909087 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:06.910173 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:06.911703 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:06.912009 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:06.912077 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:06.914871 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:06.914873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:06.914873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:06.914882 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:06.917014 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:06.917551 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:06.917552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:06.917553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:06.917684 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:06.917972 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:06.917973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:06.917973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:06.918117 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:06.918190 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:06.919127 (+ 937us) server_negotiation.cc:299] Negotiation successful
0928 13:10:06.919216 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":77,"threads_started":1}
I20260928 13:10:06.920045 20012 heartbeater.cc:347] Connected to a master server at 127.8.88.190:36031
I20260928 13:10:06.920161 20012 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:06.920369 20012 heartbeater.cc:511] Master 127.8.88.190:36031 requested a full tablet report, sending...
I20260928 13:10:06.920990 19800 ts_manager.cc:194] Registered new tserver with Master: 5228f778fcbf40008ddcb1066d6d3a99 (127.8.88.129:38349)
I20260928 13:10:06.921944 19800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:57371
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:10:06.974953 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.88.148:32979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36031
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 13:10:07 dist-test-slave-p1lc krb5kdc[19324](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790601007, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 13:10:07.121320 20020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:10:07.121716 20020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:10:07.121811 20020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:10:07.123903 20020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:10:07.123978 20020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:10:07.124023 20020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:10:07.126688 20020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.88.148:32979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:10:07.127059 20020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:10:07.127389 20020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:10:07.134984 20026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:07.134997 20029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:07.135308 20027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:07.135953 20020 server_base.cc:1034] running on GCE node
I20260928 13:10:07.136184 20020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:10:07.136454 20020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:10:07.138513 20020 hybrid_clock.cc:648] HybridClock initialized: now 1790601007138488 us; error 36 us; skew 500 ppm
Sep 28 13:10:07 dist-test-slave-p1lc krb5kdc[19324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601007, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:10:07.142110 20020 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:10:07.142673 20020 webserver.cc:460] Webserver started at http://127.8.88.130:38117/ using document root <none> and password file <none>
I20260928 13:10:07.142928 20020 fs_manager.cc:359] Metadata directory not provided
I20260928 13:10:07.142978 20020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:10:07.143105 20020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 13:10:07 dist-test-slave-p1lc krb5kdc[19324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601007, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:07.159369 20020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "32fce783a91e4dbabfc38f26cffaaf9a"
format_stamp: "Formatted at 2026-09-28 13:10:07 on dist-test-slave-p1lc"
server_key: "28ddf2a735779b2a6055e17a44ba9cf8"
server_key_iv: "2e92d41af5bc5a20949ff91bd8a3ca21"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:07.160035 20020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "32fce783a91e4dbabfc38f26cffaaf9a"
format_stamp: "Formatted at 2026-09-28 13:10:07 on dist-test-slave-p1lc"
server_key: "28ddf2a735779b2a6055e17a44ba9cf8"
server_key_iv: "2e92d41af5bc5a20949ff91bd8a3ca21"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:07.162212 20020 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 13:10:07.176167 20036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:07.176386 20020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 13:10:07.176476 20020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "32fce783a91e4dbabfc38f26cffaaf9a"
format_stamp: "Formatted at 2026-09-28 13:10:07 on dist-test-slave-p1lc"
server_key: "28ddf2a735779b2a6055e17a44ba9cf8"
server_key_iv: "2e92d41af5bc5a20949ff91bd8a3ca21"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:07.176585 20020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:10:07.198712 20020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:10:07.201293 20020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:10:07.201501 20020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:10:07.201870 20020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:10:07.202258 20020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:10:07.202337 20020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:07.202405 20020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:10:07.202459 20020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:07.209450 20020 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:39455
I20260928 13:10:07.210176 20020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:10:07.211827 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 20020
I20260928 13:10:07.211931 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:10:07.212119 8546 external_mini_cluster.cc:1428] Setting key 02f7d88d1f5db1004a7fcb506e90b6d2
I20260928 13:10:07.213724 20149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:39455 every 8 connection(s)
Sep 28 13:10:07 dist-test-slave-p1lc krb5kdc[19324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601007, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:10:07.231849 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.215874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:47195 (local address 127.8.88.190:36031)
0928 13:10:07.215935 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:07.215937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:07.217283 (+ 1346us) server_negotiation.cc:407] Connection header received
0928 13:10:07.218023 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:07.218024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:07.218041 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:07.218072 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:07.221020 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.221757 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.223539 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.223821 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.223867 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.226681 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:07.226682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:07.226683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:07.226691 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:07.230187 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:07.230734 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:07.230735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:07.230736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:07.230798 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:07.231065 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:07.231066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:07.231067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:07.231211 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:07.231260 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:07.231632 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 13:10:07.231726 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 13:10:07.232646 20153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.215388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:36031 (local address 127.8.88.130:47195)
0928 13:10:07.217032 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:07.217086 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:07.217893 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:07.218267 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:07.218272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:07.218532 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:10:07.218821 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.218827 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.221893 (+ 3066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.221893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.223191 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.223195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.223970 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.223971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.224121 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.224787 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:10:07.224797 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:10:07.226561 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:10:07.230294 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:07.230305 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:07.230308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:07.230640 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:07.230864 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:07.230865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:07.230865 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:07.230993 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:07.231291 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:10:07.231298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:10:07.231468 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 13:10:07.232213 (+ 745us) client_negotiation.cc:240] Negotiation successful
0928 13:10:07.232392 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1588,"thread_start_us":170,"threads_started":1}
I20260928 13:10:07.233394 20150 heartbeater.cc:347] Connected to a master server at 127.8.88.190:36031
I20260928 13:10:07.233490 20150 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:07.233697 20150 heartbeater.cc:511] Master 127.8.88.190:36031 requested a full tablet report, sending...
I20260928 13:10:07.234105 19800 ts_manager.cc:194] Registered new tserver with Master: 32fce783a91e4dbabfc38f26cffaaf9a (127.8.88.130:39455)
I20260928 13:10:07.234584 19800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:47195
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:10:07.273255 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.88.148:32979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36031
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:10:07.403487 20155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:10:07.403880 20155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:10:07.403973 20155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:10:07.405961 20155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:10:07.406231 20155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:10:07.406335 20155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:10:07.409025 20155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.88.148:32979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:10:07.409540 20155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:10:07.409878 20155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:10:07.417016 20161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:07.418388 20155 server_base.cc:1034] running on GCE node
W20260928 13:10:07.419729 20160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:07.423310 20163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:07.423723 20155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:10:07.424069 20155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:10:07.425238 20155 hybrid_clock.cc:648] HybridClock initialized: now 1790601007425218 us; error 37 us; skew 500 ppm
Sep 28 13:10:07 dist-test-slave-p1lc krb5kdc[19324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601007, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:10:07.428808 20155 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:10:07.429411 20155 webserver.cc:460] Webserver started at http://127.8.88.131:33977/ using document root <none> and password file <none>
I20260928 13:10:07.429669 20155 fs_manager.cc:359] Metadata directory not provided
I20260928 13:10:07.429726 20155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:10:07.429859 20155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 13:10:07 dist-test-slave-p1lc krb5kdc[19324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601007, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:07.455303 20155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "d7f09716c15c44b8bbd2b53b292fe79b"
format_stamp: "Formatted at 2026-09-28 13:10:07 on dist-test-slave-p1lc"
server_key: "612e246328069fd7f7cc5ee4ed0ce84b"
server_key_iv: "2256c1456d2537ebdbf6ff5eaf02b9e0"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:07.456039 20155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "d7f09716c15c44b8bbd2b53b292fe79b"
format_stamp: "Formatted at 2026-09-28 13:10:07 on dist-test-slave-p1lc"
server_key: "612e246328069fd7f7cc5ee4ed0ce84b"
server_key_iv: "2256c1456d2537ebdbf6ff5eaf02b9e0"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:07.473347 20155 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.000s sys 0.002s
I20260928 13:10:07.493742 20170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:07.494029 20155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:10:07.494129 20155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "d7f09716c15c44b8bbd2b53b292fe79b"
format_stamp: "Formatted at 2026-09-28 13:10:07 on dist-test-slave-p1lc"
server_key: "612e246328069fd7f7cc5ee4ed0ce84b"
server_key_iv: "2256c1456d2537ebdbf6ff5eaf02b9e0"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:07.494230 20155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:10:07.520015 20155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:10:07.522737 20155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:10:07.522943 20155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:10:07.523249 20155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:10:07.523701 20155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:10:07.523794 20155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:07.523875 20155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:10:07.523936 20155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:07.531355 20155 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:36469
I20260928 13:10:07.531905 20155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:10:07.534925 20283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:36469 every 8 connection(s)
I20260928 13:10:07.541589 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 20155
I20260928 13:10:07.541705 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:10:07.541934 8546 external_mini_cluster.cc:1428] Setting key 4b040e49022cb5fddde674cec726c261
Sep 28 13:10:07 dist-test-slave-p1lc krb5kdc[19324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601007, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:10:07.553565 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.536318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:43383 (local address 127.8.88.190:36031)
0928 13:10:07.536637 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:07.536639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:07.536652 (+ 13us) server_negotiation.cc:407] Connection header received
0928 13:10:07.537381 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:07.537382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:07.537395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:07.537441 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:07.538335 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.539118 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.540748 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.541096 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.541141 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.544514 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:07.544516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:07.544516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:07.544525 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:07.549874 (+ 5349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:07.551710 (+ 1836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:07.551713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:07.551713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:07.551793 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:07.552158 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:07.552159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:07.552159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:07.552320 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:07.552608 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:07.553339 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 13:10:07.553446 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
I20260928 13:10:07.555042 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.536288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:36031 (local address 127.8.88.131:43383)
0928 13:10:07.536468 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:07.536527 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:07.537242 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:07.537650 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:07.537663 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:07.537917 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:10:07.538213 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.538220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.539243 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.539244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.540643 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.540646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.541239 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.541241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.541388 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.542332 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:10:07.542342 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:10:07.544386 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:10:07.550081 (+ 5695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:07.550094 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:07.550098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:07.551580 (+ 1482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:07.551879 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:07.551880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:07.551880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:07.552062 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:07.552665 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:10:07.552674 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:10:07.553142 (+ 468us) client_negotiation.cc:769] Sending connection context
0928 13:10:07.554498 (+ 1356us) client_negotiation.cc:240] Negotiation successful
0928 13:10:07.554740 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260928 13:10:07.556360 20284 heartbeater.cc:347] Connected to a master server at 127.8.88.190:36031
I20260928 13:10:07.556488 20284 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:07.556977 20284 heartbeater.cc:511] Master 127.8.88.190:36031 requested a full tablet report, sending...
I20260928 13:10:07.559095 19800 ts_manager.cc:194] Registered new tserver with Master: d7f09716c15c44b8bbd2b53b292fe79b (127.8.88.131:36469)
I20260928 13:10:07.559676 19800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:43383
I20260928 13:10:07.570039 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:10:07.599627 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.577271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.8.88.190:36031)
0928 13:10:07.581208 (+ 3937us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:07.581211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:07.589121 (+ 7910us) server_negotiation.cc:407] Connection header received
0928 13:10:07.589161 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:07.589163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:07.589178 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:07.589241 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:07.592465 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.593267 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.594372 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.594651 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.594718 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.596375 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:07.596376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:07.596377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:07.596385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:07.598207 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:07.598626 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:07.598628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:07.598628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:07.598722 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:07.598975 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:07.598975 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:07.598976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:07.599127 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:07.599208 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:07.599430 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 13:10:07.599532 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3892}
I20260928 13:10:07.603994 19800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:10:07.606490 19800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 28 13:10:07 dist-test-slave-p1lc krb5kdc[19324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601007, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:07.638907 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.623677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:36469 (local address 127.0.0.1:35088)
0928 13:10:07.628366 (+ 4689us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:07.628374 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:07.628410 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:07.628741 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:07.628745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:07.628747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:10:07.628833 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.628835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.630209 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.630211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.631574 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.631578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.638549 (+ 6971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.638549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.638659 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.638678 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 13:10:07.638743 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 13:10:07.638808 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4678,"thread_start_us":75,"threads_started":1}
I20260928 13:10:07.640502 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.624120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35088 (local address 127.8.88.131:36469)
0928 13:10:07.624298 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:07.624305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:07.628470 (+ 4165us) server_negotiation.cc:407] Connection header received
0928 13:10:07.628526 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:07.628529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:07.628547 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:07.628621 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:10:07.628946 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.630095 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.636969 (+ 6874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.637850 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.637943 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.638029 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:07.640348 (+ 2319us) server_negotiation.cc:299] Negotiation successful
0928 13:10:07.640408 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260928 13:10:07.641332 20216 tablet_service.cc:1467] Processing CreateTablet for tablet 29d75d6b764b45f28e0bbca706265ab3 (DEFAULT_TABLE table=test-table [id=c90b2512140e47cea8b0c966d14d557e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:10:07.641758 20216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29d75d6b764b45f28e0bbca706265ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:10:07.645546 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.623418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:39455 (local address 127.0.0.1:43282)
0928 13:10:07.627998 (+ 4580us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:07.628062 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:07.628152 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:07.638414 (+ 10262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:07.638416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:07.638419 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:10:07.638476 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.638478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.641117 (+ 2639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.641119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.642294 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.642297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.645295 (+ 2998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.645297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.645378 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.645384 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 13:10:07.645414 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 13:10:07.645463 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4535,"thread_start_us":105,"threads_started":1}
I20260928 13:10:07.647387 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.623819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43282 (local address 127.8.88.130:39455)
0928 13:10:07.638158 (+ 14339us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:07.638164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:07.638175 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:10:07.638219 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:07.638221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:07.638239 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:07.638317 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:10:07.639037 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.640222 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.642429 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.643700 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.647078 (+ 3378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.647207 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:07.647239 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 13:10:07.647296 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14277,"thread_start_us":79,"threads_started":1}
I20260928 13:10:07.648972 20084 tablet_service.cc:1467] Processing CreateTablet for tablet 29d75d6b764b45f28e0bbca706265ab3 (DEFAULT_TABLE table=test-table [id=c90b2512140e47cea8b0c966d14d557e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:10:07.649343 20084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29d75d6b764b45f28e0bbca706265ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:10:07.653445 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.623422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:38349 (local address 127.0.0.1:49850)
0928 13:10:07.628263 (+ 4841us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:07.628271 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:07.628303 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:07.643825 (+ 15522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:07.643826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:07.643830 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:10:07.643879 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.643880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.645682 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.645683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.646985 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.646988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.653179 (+ 6191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.653181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.653268 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.653274 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 13:10:07.653305 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 13:10:07.653358 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4824,"thread_start_us":132,"threads_started":1}
I20260928 13:10:07.653972 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.623794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.8.88.129:38349)
0928 13:10:07.640910 (+ 17116us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:07.640916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:07.640929 (+ 13us) server_negotiation.cc:407] Connection header received
0928 13:10:07.640969 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:07.640972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:07.640992 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:07.643960 (+ 2968us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:10:07.644034 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.645178 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.647562 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.653041 (+ 5479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.653728 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.653820 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:07.653850 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 13:10:07.653903 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17044,"thread_start_us":50,"threads_started":1}
I20260928 13:10:07.654743 19946 tablet_service.cc:1467] Processing CreateTablet for tablet 29d75d6b764b45f28e0bbca706265ab3 (DEFAULT_TABLE table=test-table [id=c90b2512140e47cea8b0c966d14d557e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:10:07.655129 19946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29d75d6b764b45f28e0bbca706265ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:10:07.660945 20302 tablet_bootstrap.cc:492] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b: Bootstrap starting.
I20260928 13:10:07.663909 20302 tablet_bootstrap.cc:654] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b: Neither blocks nor log segments found. Creating new log.
I20260928 13:10:07.665089 20303 tablet_bootstrap.cc:492] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a: Bootstrap starting.
I20260928 13:10:07.665536 20304 tablet_bootstrap.cc:492] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99: Bootstrap starting.
I20260928 13:10:07.666323 20303 tablet_bootstrap.cc:654] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a: Neither blocks nor log segments found. Creating new log.
I20260928 13:10:07.666364 20304 tablet_bootstrap.cc:654] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99: Neither blocks nor log segments found. Creating new log.
I20260928 13:10:07.666761 20303 log.cc:824] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a: Log is configured to *not* fsync() on all Append() calls
I20260928 13:10:07.666761 20304 log.cc:824] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99: Log is configured to *not* fsync() on all Append() calls
I20260928 13:10:07.667982 20302 log.cc:824] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:10:07.674427 20303 tablet_bootstrap.cc:492] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a: No bootstrap required, opened a new log
I20260928 13:10:07.674543 20303 ts_tablet_manager.cc:1397] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260928 13:10:07.675200 20304 tablet_bootstrap.cc:492] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99: No bootstrap required, opened a new log
I20260928 13:10:07.675448 20304 ts_tablet_manager.cc:1397] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260928 13:10:07.676113 20302 tablet_bootstrap.cc:492] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b: No bootstrap required, opened a new log
I20260928 13:10:07.676182 20302 ts_tablet_manager.cc:1397] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20260928 13:10:07.676890 20303 raft_consensus.cc:348] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5228f778fcbf40008ddcb1066d6d3a99" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 38349 } } peers { permanent_uuid: "32fce783a91e4dbabfc38f26cffaaf9a" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39455 } } peers { permanent_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 36469 } }
I20260928 13:10:07.677091 20303 raft_consensus.cc:374] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:10:07.677130 20303 raft_consensus.cc:729] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32fce783a91e4dbabfc38f26cffaaf9a, State: Initialized, Role: FOLLOWER
I20260928 13:10:07.677534 20303 consensus_queue.cc:260] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5228f778fcbf40008ddcb1066d6d3a99" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 38349 } } peers { permanent_uuid: "32fce783a91e4dbabfc38f26cffaaf9a" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39455 } } peers { permanent_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 36469 } }
I20260928 13:10:07.677945 20303 ts_tablet_manager.cc:1428] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:10:07.678133 20150 heartbeater.cc:503] Master 127.8.88.190:36031 was elected leader, sending a full tablet report...
I20260928 13:10:07.679662 20302 raft_consensus.cc:348] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5228f778fcbf40008ddcb1066d6d3a99" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 38349 } } peers { permanent_uuid: "32fce783a91e4dbabfc38f26cffaaf9a" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39455 } } peers { permanent_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 36469 } }
I20260928 13:10:07.679833 20304 raft_consensus.cc:348] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5228f778fcbf40008ddcb1066d6d3a99" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 38349 } } peers { permanent_uuid: "32fce783a91e4dbabfc38f26cffaaf9a" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39455 } } peers { permanent_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 36469 } }
I20260928 13:10:07.680083 20304 raft_consensus.cc:374] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:10:07.680214 20302 raft_consensus.cc:374] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:10:07.680341 20302 raft_consensus.cc:729] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f09716c15c44b8bbd2b53b292fe79b, State: Initialized, Role: FOLLOWER
I20260928 13:10:07.680163 20304 raft_consensus.cc:729] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5228f778fcbf40008ddcb1066d6d3a99, State: Initialized, Role: FOLLOWER
I20260928 13:10:07.680490 20304 consensus_queue.cc:260] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5228f778fcbf40008ddcb1066d6d3a99" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 38349 } } peers { permanent_uuid: "32fce783a91e4dbabfc38f26cffaaf9a" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39455 } } peers { permanent_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 36469 } }
I20260928 13:10:07.680958 20302 consensus_queue.cc:260] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5228f778fcbf40008ddcb1066d6d3a99" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 38349 } } peers { permanent_uuid: "32fce783a91e4dbabfc38f26cffaaf9a" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39455 } } peers { permanent_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 36469 } }
I20260928 13:10:07.681324 20302 ts_tablet_manager.cc:1428] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 13:10:07.681382 20304 ts_tablet_manager.cc:1428] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260928 13:10:07.681591 20012 heartbeater.cc:503] Master 127.8.88.190:36031 was elected leader, sending a full tablet report...
I20260928 13:10:07.682461 20284 heartbeater.cc:503] Master 127.8.88.190:36031 was elected leader, sending a full tablet report...
W20260928 13:10:07.714561 20151 tablet.cc:2369] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:10:07.786291 20285 tablet.cc:2369] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:10:07.895658 20310 raft_consensus.cc:482] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:10:07.895815 20310 raft_consensus.cc:504] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5228f778fcbf40008ddcb1066d6d3a99" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 38349 } } peers { permanent_uuid: "32fce783a91e4dbabfc38f26cffaaf9a" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39455 } } peers { permanent_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 36469 } }
I20260928 13:10:07.896207 20310 leader_election.cc:288] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5228f778fcbf40008ddcb1066d6d3a99 (127.8.88.129:38349), 32fce783a91e4dbabfc38f26cffaaf9a (127.8.88.130:39455)
I20260928 13:10:07.901168 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.896489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:38349 (local address 127.8.88.131:35407)
0928 13:10:07.896938 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:07.896952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:07.897005 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:07.897793 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:07.897794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:07.897797 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:10:07.897870 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.897872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.898840 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.898841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.900082 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.900085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.900918 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.900919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.900993 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.900997 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:10:07.901043 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 13:10:07.901087 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427}
I20260928 13:10:07.901932 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.897163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:35407 (local address 127.8.88.129:38349)
0928 13:10:07.897259 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:07.897261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:07.897271 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:10:07.897301 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:07.897303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:07.897316 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:07.897388 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:10:07.897965 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.898709 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.900209 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.900778 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.900848 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.900857 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:07.901818 (+ 961us) server_negotiation.cc:299] Negotiation successful
0928 13:10:07.901868 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 13:10:07.902184 19963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d75d6b764b45f28e0bbca706265ab3" candidate_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5228f778fcbf40008ddcb1066d6d3a99" is_pre_election: true
I20260928 13:10:07.902344 19963 raft_consensus.cc:2454] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f09716c15c44b8bbd2b53b292fe79b in term 0.
I20260928 13:10:07.902875 20172 leader_election.cc:302] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5228f778fcbf40008ddcb1066d6d3a99, d7f09716c15c44b8bbd2b53b292fe79b; no voters:
I20260928 13:10:07.903069 20310 raft_consensus.cc:2781] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:10:07.903117 20310 raft_consensus.cc:482] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:10:07.903137 20310 raft_consensus.cc:3037] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:10:07.904037 20310 raft_consensus.cc:504] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5228f778fcbf40008ddcb1066d6d3a99" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 38349 } } peers { permanent_uuid: "32fce783a91e4dbabfc38f26cffaaf9a" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39455 } } peers { permanent_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 36469 } }
I20260928 13:10:07.904168 20310 leader_election.cc:288] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [CANDIDATE]: Term 1 election: Requested vote from peers 5228f778fcbf40008ddcb1066d6d3a99 (127.8.88.129:38349), 32fce783a91e4dbabfc38f26cffaaf9a (127.8.88.130:39455)
I20260928 13:10:07.904462 19963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d75d6b764b45f28e0bbca706265ab3" candidate_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5228f778fcbf40008ddcb1066d6d3a99"
I20260928 13:10:07.904536 19963 raft_consensus.cc:3037] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:10:07.905509 19963 raft_consensus.cc:2454] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f09716c15c44b8bbd2b53b292fe79b in term 1.
I20260928 13:10:07.905745 20172 leader_election.cc:302] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5228f778fcbf40008ddcb1066d6d3a99, d7f09716c15c44b8bbd2b53b292fe79b; no voters:
I20260928 13:10:07.905860 20310 raft_consensus.cc:2781] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:10:07.906019 20310 raft_consensus.cc:686] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [term 1 LEADER]: Becoming Leader. State: Replica: d7f09716c15c44b8bbd2b53b292fe79b, State: Running, Role: LEADER
I20260928 13:10:07.906112 20310 consensus_queue.cc:237] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5228f778fcbf40008ddcb1066d6d3a99" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 38349 } } peers { permanent_uuid: "32fce783a91e4dbabfc38f26cffaaf9a" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39455 } } peers { permanent_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 36469 } }
I20260928 13:10:07.906976 19799 catalog_manager.cc:5495] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b reported cstate change: term changed from 0 to 1, leader changed from <none> to d7f09716c15c44b8bbd2b53b292fe79b (127.8.88.131). New cstate: current_term: 1 leader_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5228f778fcbf40008ddcb1066d6d3a99" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 38349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32fce783a91e4dbabfc38f26cffaaf9a" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 36469 } health_report { overall_health: HEALTHY } } }
W20260928 13:10:07.909818 20013 tablet.cc:2369] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:10:07.916240 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.896611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:39455 (local address 127.8.88.131:40865)
0928 13:10:07.901259 (+ 4648us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:07.901271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:07.901311 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:07.910604 (+ 9293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:07.910619 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:07.910623 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:10:07.910681 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.910683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.913989 (+ 3306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.913992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.915147 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:07.915151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.915985 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.915987 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:07.916055 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.916059 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:10:07.916101 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 13:10:07.916150 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4636,"thread_start_us":105,"threads_started":1}
I20260928 13:10:07.916704 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.897150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:40865 (local address 127.8.88.130:39455)
0928 13:10:07.897212 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:07.897214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:07.901365 (+ 4151us) server_negotiation.cc:407] Connection header received
0928 13:10:07.901396 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:07.901397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:07.901407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:07.901447 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:10:07.911067 (+ 9620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.911844 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.915283 (+ 3439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.915858 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.915936 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.915946 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:07.916592 (+ 646us) server_negotiation.cc:299] Negotiation successful
0928 13:10:07.916639 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 13:10:07.917315 20088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d75d6b764b45f28e0bbca706265ab3" candidate_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fce783a91e4dbabfc38f26cffaaf9a" is_pre_election: true
I20260928 13:10:07.917505 20088 raft_consensus.cc:2454] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f09716c15c44b8bbd2b53b292fe79b in term 0.
I20260928 13:10:07.917625 20101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29d75d6b764b45f28e0bbca706265ab3" candidate_uuid: "d7f09716c15c44b8bbd2b53b292fe79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32fce783a91e4dbabfc38f26cffaaf9a"
I20260928 13:10:07.917681 20101 raft_consensus.cc:3037] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:10:07.918588 20101 raft_consensus.cc:2454] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f09716c15c44b8bbd2b53b292fe79b in term 1.
I20260928 13:10:07.922600 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.918472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35100 (local address 127.8.88.131:36469)
0928 13:10:07.918532 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:07.918534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:07.918546 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:10:07.918584 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:07.918585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:07.918609 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:07.918706 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:10:07.919058 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.919798 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.920923 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:07.921676 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:07.921750 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:07.921978 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:10:07.922118 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:10:07.922200 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:07.922458 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 13:10:07.922481 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 13:10:07.924957 20101 raft_consensus.cc:1261] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a [term 1 FOLLOWER]: Refusing update from remote peer d7f09716c15c44b8bbd2b53b292fe79b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:10:07.925395 20312 consensus_queue.cc:1035] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32fce783a91e4dbabfc38f26cffaaf9a" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:10:07.927691 19963 raft_consensus.cc:1261] T 29d75d6b764b45f28e0bbca706265ab3 P 5228f778fcbf40008ddcb1066d6d3a99 [term 1 FOLLOWER]: Refusing update from remote peer d7f09716c15c44b8bbd2b53b292fe79b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:10:07.928056 20312 consensus_queue.cc:1035] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5228f778fcbf40008ddcb1066d6d3a99" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:10:07.932480 20315 mvcc.cc:205] Tried to move back new op lower bound from 7334301728457232384 to 7334301728383905792. Current Snapshot: MvccSnapshot[applied={T|T < 7334301728457232384}]
I20260928 13:10:07.933157 20316 mvcc.cc:205] Tried to move back new op lower bound from 7334301728457232384 to 7334301728383905792. Current Snapshot: MvccSnapshot[applied={T|T < 7334301728457232384}]
I20260928 13:10:07.934841 20318 mvcc.cc:205] Tried to move back new op lower bound from 7334301728457232384 to 7334301728383905792. Current Snapshot: MvccSnapshot[applied={T|T < 7334301728457232384}]
I20260928 13:10:07.937168 19799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:10:07.937261 19799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:10:07.938098 19799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b369f00fa684b2e80dc508c6ca5ffc9: Sending DeleteTablet for 3 replicas of tablet 29d75d6b764b45f28e0bbca706265ab3
I20260928 13:10:07.938426 19946 tablet_service.cc:1514] Processing DeleteTablet for tablet 29d75d6b764b45f28e0bbca706265ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:10:07 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:49850
I20260928 13:10:07.938915 20084 tablet_service.cc:1514] Processing DeleteTablet for tablet 29d75d6b764b45f28e0bbca706265ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:10:07 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:43282
I20260928 13:10:07.939097 20216 tablet_service.cc:1514] Processing DeleteTablet for tablet 29d75d6b764b45f28e0bbca706265ab3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:10:07 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:35088
I20260928 13:10:07.939718 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 19881
I20260928 13:10:07.944294 20326 tablet_replica.cc:330] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b: stopping tablet replica
I20260928 13:10:07.944482 20326 raft_consensus.cc:2229] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:10:07.944764 20326 raft_consensus.cc:2258] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:10:07.945029 20325 tablet_replica.cc:330] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a: stopping tablet replica
I20260928 13:10:07.945261 20325 raft_consensus.cc:2229] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:10:07.945408 20325 raft_consensus.cc:2258] T 29d75d6b764b45f28e0bbca706265ab3 P 32fce783a91e4dbabfc38f26cffaaf9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:10:07.953502 20326 ts_tablet_manager.cc:1905] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 13:10:07.955430 19785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:38349 (error 108)
I20260928 13:10:07.956223 20326 ts_tablet_manager.cc:1918] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:10:07.956414 20326 log.cc:1196] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/29d75d6b764b45f28e0bbca706265ab3
I20260928 13:10:07.956949 20326 ts_tablet_manager.cc:1939] T 29d75d6b764b45f28e0bbca706265ab3 P d7f09716c15c44b8bbd2b53b292fe79b: Deleting consensus metadata
I20260928 13:10:07.957533 19786 catalog_manager.cc:4841] TS d7f09716c15c44b8bbd2b53b292fe79b (127.8.88.131:36469): tablet 29d75d6b764b45f28e0bbca706265ab3 (table test-table [id=c90b2512140e47cea8b0c966d14d557e]) successfully deleted
I20260928 13:10:07.957831 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 20020
I20260928 13:10:07.960528 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.960361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:38349 (local address 127.0.0.1:49860)
0928 13:10:07.960421 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:07.960482 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:38349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 13:10:07.960772 19785 catalog_manager.cc:4584] TS 5228f778fcbf40008ddcb1066d6d3a99 (127.8.88.129:38349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29d75d6b764b45f28e0bbca706265ab3: Network error: Client connection negotiation failed: client connection to 127.8.88.129:38349: connect: Connection refused (error 111)
I20260928 13:10:07.963709 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:07.963576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:39455 (local address 127.0.0.1:43298)
0928 13:10:07.963627 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:07.963665 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.130:39455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 13:10:07.964960 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 20155
I20260928 13:10:07.972905 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 19768
I20260928 13:10:07.978746 8546 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 13:10:08.314162 8546 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 13:10:08.314261 8546 mini_ranger.cc:66] Stopping Ranger...
I20260928 13:10:08.659816 8546 mini_ranger.cc:68] Stopped Ranger
2026-09-28 13:10:08.660 UTC [19354] LOG: received smart shutdown request
2026-09-28 13:10:08.663 UTC [19354] LOG: background worker "logical replication launcher" (PID 19363) exited with exit code 1
2026-09-28 13:10:08.664 UTC [19358] LOG: shutting down
2026-09-28 13:10:08.877 UTC [19354] LOG: database system is shut down
2026-09-28T13:10:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50024 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:10:09 dist-test-slave-p1lc krb5kdc[20336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:10:09 dist-test-slave-p1lc krb5kdc[20336](info): set up 2 sockets
Sep 28 13:10:09 dist-test-slave-p1lc krb5kdc[20336](info): commencing operation
krb5kdc: starting...
W20260928 13:10:10.037401 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:10:10 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:10:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:10:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.88.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.88.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/rangeradmin_127.8.88.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/rangeradmin_127.8.88.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.88.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.88.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/rangerlookup_127.8.88.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/rangerlookup_127.8.88.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.88.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.88.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/HTTP_127.8.88.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/HTTP_127.8.88.148@KRBTEST.COM.keytab.
I20260928 13:10:10.208367 8546 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcZlb_Q/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/postgres -l logfile start
2026-09-28 13:10:13.063 UTC [20366] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 13:10:13.063 UTC [20366] LOG: listening on IPv4 address "127.8.88.148", port 39537
2026-09-28 13:10:13.068 UTC [20366] LOG: listening on Unix socket "/tmp/.s.PGSQL.39537"
2026-09-28 13:10:13.105 UTC [20369] LOG: database system was shut down at 2026-09-28 13:10:11 UTC
2026-09-28 13:10:13.110 UTC [20366] LOG: database system is ready to accept connections
I20260928 13:10:14.064827 8546 mini_postgres.cc:96] Postgres bound to 39537
2026-09-28 13:10:14.071 UTC [20377] FATAL: database "slave" does not exist
127.8.88.148:39537 - accepting connections
I20260928 13:10:14.071949 8546 mini_ranger.cc:161] Starting Ranger...
I20260928 13:10:14.088326 8546 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 13:10:14.685365 8546 mini_ranger.cc:87] Created ranger Postgres database
I20260928 13:10:14.685566 8546 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-admin
2026-09-28 13:10:14,998 [I] DB FLAVOR :POSTGRES
2026-09-28 13:10:14,999 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 13:10:14,999 [I] Checking connection..
2026-09-28 13:10:14,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 13:10:15,278 [I] Checking connection passed.
2026-09-28 13:10:15,278 [I] --------- Verifying version history table ---------
2026-09-28 13:10:15,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 13:10:15,537 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 13:10:15,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 13:10:15,811 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 13:10:15,811 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 13:10:15,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 13:10:16.060 UTC [20441] WARNING: there is no transaction in progress
2026-09-28 13:10:16,067 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 13:10:16,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 13:10:16,314 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 13:10:16,314 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 13:10:16,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 13:10:16,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 13:10:16,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p1lc.c.gcp-upstream.internal','N') ;"
2026-09-28 13:10:17,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 13:10:17,307 [I] Table x_portal_user does not exist in database ranger
2026-09-28 13:10:17,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 13:10:17,576 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 13:10:17,576 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 13:10:17,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 13:10:18,101 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 13:10:18,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 13:10:19.218 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.236 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.255 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.275 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.296 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.319 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.392 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.400 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.417 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.429 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.442 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.458 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.469 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.477 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.486 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:19.496 UTC [20549] WARNING: there is no transaction in progress
2026-09-28 13:10:20,357 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 13:10:20,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 13:10:20,599 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 13:10:20,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 13:10:20,843 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 13:10:20,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 13:10:21,109 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 13:10:21,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 13:10:21,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 13:10:21,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p1lc.c.gcp-upstream.internal';"
2026-09-28 13:10:21,863 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 13:10:21,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 13:10:22,132 [I] DB_PATCHES have already been applied
I20260928 13:10:22.138366 8546 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcZlb_Q/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 13:10:22.138463 8546 mini_ranger.cc:193] Using host: 127.8.88.148
I20260928 13:10:22.141714 8546 mini_ranger.cc:227] Ranger admin URL: http://127.8.88.148:49783
Sep 28, 2026 1:10:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 1:10:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 1:10:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 1:10:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 1:10:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 1:10:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 1:10:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 13:10:23.287976 8546 mini_ranger.cc:160] Time spent starting Ranger: real 9.216s user 0.000s sys 0.009s
Sep 28, 2026 1:10:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 1:10:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 1:10:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 1:10:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 1:10:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 1:10:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 1:10:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 1:10:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 1:10:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 1:10:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 1:10:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 1:10:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 1:10:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 1:10:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 1:10:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 1:10:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 1:10:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 1:10:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 13:10:45.319168 8546 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.88.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.88.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/rangerkms_127.8.88.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.88.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/rangerkms_127.8.88.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.88.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.88.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.88.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/HTTP_127.8.88.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.88.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/HTTP_127.8.88.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 13:10:45 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601045, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 13:10:45.407073 8546 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 13:10:45.417110 8546 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 13:10:46.487162 8546 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 13:10:46.487248 8546 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-kms
I20260928 13:10:46.487270 8546 mini_ranger_kms.cc:227] Using postgres at 127.8.88.148:39537
2026-09-28 13:10:46,653 [I] DB FLAVOR :POSTGRES
2026-09-28 13:10:46,653 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 13:10:46,653 [I] Checking connection
2026-09-28 13:10:46,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 13:10:46,894 [I] connection success
2026-09-28 13:10:46,894 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 13:10:46,894 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 13:10:46,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 13:10:47,133 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 13:10:47,133 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 13:10:47,134 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 13:10:47,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.88.148:39537/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 13:10:47,392 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 13:10:47.902535 8546 mini_ranger_kms.cc:310] Created kms service
I20260928 13:10:48.095674 8546 mini_ranger_kms.cc:326] Created kudu user
I20260928 13:10:48.142856 8546 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 13:10:48 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601045, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:48.562041 8546 mini_ranger_kms.cc:384] Added ranger policy
I20260928 13:10:48.562160 8546 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcZlb_Q/build/release/bin/postgresql.jar:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcZlb_Q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 13:10:48.562181 8546 mini_ranger_kms.cc:240] Using host: 127.8.88.148
I20260928 13:10:48.565874 8546 mini_ranger_kms.cc:276] Ranger KMS PID: 20729
I20260928 13:10:48.565964 8546 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.88.148:34603
Sep 28, 2026 1:10:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 1:10:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 1:10:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 1:10:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 1:10:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 1:10:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 13:10:50.880591 8546 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.474s user 0.004s sys 0.144s
I20260928 13:10:50.880705 8546 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 13:10:50.880750 8546 mini_ranger_kms.cc:401] 127.8.88.148:34603/kms/v1/keys
Sep 28 13:10:51 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790601051, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.88.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 13:10:51 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601051, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.88.148@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
Sep 28 13:10:52 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601051, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.88.148@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
Sep 28 13:10:54 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:10:54.431854 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46263
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46263
--ranger_config_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.88.148:34603/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:10:54.529307 20780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:10:54.529525 20780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:10:54.529548 20780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:10:54.530769 20780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:10:54.530818 20780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:10:54.530833 20780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:10:54.530845 20780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:10:54.530857 20780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:10:54.532549 20780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.88.148:34603/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46263
--ranger_config_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:10:54.532814 20780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:10:54.533090 20780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:10:54.539825 20786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:54.539825 20788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:54.539968 20785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:54.539986 20780 server_base.cc:1034] running on GCE node
I20260928 13:10:54.540356 20780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:10:54.540676 20780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:10:54.541877 20780 hybrid_clock.cc:648] HybridClock initialized: now 1790601054541849 us; error 42 us; skew 500 ppm
Sep 28 13:10:54 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601054, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:10:54.551242 20780 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:10:54.551864 20780 webserver.cc:460] Webserver started at http://127.8.88.190:46387/ using document root <none> and password file <none>
I20260928 13:10:54.552140 20780 fs_manager.cc:359] Metadata directory not provided
I20260928 13:10:54.552213 20780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:10:54.552376 20780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 13:10:54 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601054, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:54.599192 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "01017f7c9e5f4ed48e150b049ce0c458"
format_stamp: "Formatted at 2026-09-28 13:10:54 on dist-test-slave-p1lc"
server_key: "fc65ea7a56946ee231ee0b8a5b6217b1"
server_key_iv: "20c391d89fbe7ce34c11dbc52275acb2"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:54.599875 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "01017f7c9e5f4ed48e150b049ce0c458"
format_stamp: "Formatted at 2026-09-28 13:10:54 on dist-test-slave-p1lc"
server_key: "fc65ea7a56946ee231ee0b8a5b6217b1"
server_key_iv: "20c391d89fbe7ce34c11dbc52275acb2"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:54.601521 20780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 13:10:54.614018 20795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:54.614284 20780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:10:54.614393 20780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "01017f7c9e5f4ed48e150b049ce0c458"
format_stamp: "Formatted at 2026-09-28 13:10:54 on dist-test-slave-p1lc"
server_key: "fc65ea7a56946ee231ee0b8a5b6217b1"
server_key_iv: "20c391d89fbe7ce34c11dbc52275acb2"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:54.614516 20780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:10:54.628958 20780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:10:54.631174 20780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:10:54.631361 20780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:10:54.636380 20780 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:46263
I20260928 13:10:54.636457 20847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:46263 every 8 connection(s)
I20260928 13:10:54.636837 20780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:10:54.637689 20848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:10:54.638995 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 20780
I20260928 13:10:54.639110 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:10:54.639307 8546 external_mini_cluster.cc:1428] Setting key d64fc0507cbe44c81bc421a071483d9b
I20260928 13:10:54.640432 20848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458: Bootstrap starting.
I20260928 13:10:54.641212 20848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458: Neither blocks nor log segments found. Creating new log.
I20260928 13:10:54.641572 20848 log.cc:824] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458: Log is configured to *not* fsync() on all Append() calls
I20260928 13:10:54.642319 20848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458: No bootstrap required, opened a new log
I20260928 13:10:54.644552 20848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01017f7c9e5f4ed48e150b049ce0c458" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46263 } }
I20260928 13:10:54.644718 20848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:10:54.644783 20848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01017f7c9e5f4ed48e150b049ce0c458, State: Initialized, Role: FOLLOWER
I20260928 13:10:54.644958 20848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01017f7c9e5f4ed48e150b049ce0c458" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46263 } }
I20260928 13:10:54.645058 20848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:10:54.645097 20848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:10:54.645176 20848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:10:54.645903 20848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01017f7c9e5f4ed48e150b049ce0c458" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46263 } }
I20260928 13:10:54.646021 20848 leader_election.cc:302] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01017f7c9e5f4ed48e150b049ce0c458; no voters:
I20260928 13:10:54.646198 20848 leader_election.cc:288] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:10:54.646255 20853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:10:54.646473 20853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 1 LEADER]: Becoming Leader. State: Replica: 01017f7c9e5f4ed48e150b049ce0c458, State: Running, Role: LEADER
I20260928 13:10:54.646581 20848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:10:54.646602 20853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01017f7c9e5f4ed48e150b049ce0c458" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46263 } }
Sep 28 13:10:54 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:10:54.647737 20853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01017f7c9e5f4ed48e150b049ce0c458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01017f7c9e5f4ed48e150b049ce0c458" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46263 } } }
I20260928 13:10:54.647833 20853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [sys.catalog]: This master's current role is: LEADER
I20260928 13:10:54.648164 20848 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 13:10:54.648248 20854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01017f7c9e5f4ed48e150b049ce0c458. Latest consensus state: current_term: 1 leader_uuid: "01017f7c9e5f4ed48e150b049ce0c458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01017f7c9e5f4ed48e150b049ce0c458" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46263 } } }
I20260928 13:10:54.648730 20854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [sys.catalog]: This master's current role is: LEADER
I20260928 13:10:54.648955 20867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:10:54.649466 20867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:10:54.653870 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:54.639856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.8.88.190:46263)
0928 13:10:54.640040 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:54.640050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:54.640081 (+ 31us) server_negotiation.cc:407] Connection header received
0928 13:10:54.640588 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:54.640597 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:54.640661 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:54.640815 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:54.641680 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:54.642984 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:54.644130 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:54.644736 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:54.644841 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:54.647936 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:54.647940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:54.647940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:54.647952 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:54.649989 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:54.650391 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:54.650394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:54.650394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:54.650492 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:54.650730 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:54.650731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:54.650731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:54.652710 (+ 1979us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:54.653015 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:54.653114 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 13:10:54.653508 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":68,"threads_started":1}
I20260928 13:10:54.655671 20867 catalog_manager.cc:1347] Generated new cluster ID: 80bb545777eb416892d7af6e18caa6be
I20260928 13:10:54.655764 20867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:10:54.664723 20867 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:10:54.665599 20867 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:10:54.672425 20867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458: Generated new TSK 0
I20260928 13:10:54.673032 20867 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:10:55.469389 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.88.148:34603/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46263
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:10:55.590593 20893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:10:55.590833 20893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:10:55.590859 20893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:10:55.592202 20893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:10:55.592264 20893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:10:55.592295 20893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:10:55.594214 20893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.88.148:34603/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:10:55.594544 20893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:10:55.594890 20893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:10:55.601019 20898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:55.601346 20899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:55.602044 20901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:55.603123 20893 server_base.cc:1034] running on GCE node
I20260928 13:10:55.603381 20893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:10:55.603735 20893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:10:55.604939 20893 hybrid_clock.cc:648] HybridClock initialized: now 1790601055604912 us; error 44 us; skew 500 ppm
Sep 28 13:10:55 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:10:55.608716 20893 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:10:55.609432 20893 webserver.cc:460] Webserver started at http://127.8.88.129:35665/ using document root <none> and password file <none>
I20260928 13:10:55.609766 20893 fs_manager.cc:359] Metadata directory not provided
I20260928 13:10:55.609869 20893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:10:55.610080 20893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 13:10:55 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:55.625867 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "710e390f949e475da4fa7c8eb1a71d25"
format_stamp: "Formatted at 2026-09-28 13:10:55 on dist-test-slave-p1lc"
server_key: "0c08d3edbb087c31569268ee390ff2e4"
server_key_iv: "de407b54b78ec39df3bac93aa7ad0722"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:55.626389 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "710e390f949e475da4fa7c8eb1a71d25"
format_stamp: "Formatted at 2026-09-28 13:10:55 on dist-test-slave-p1lc"
server_key: "0c08d3edbb087c31569268ee390ff2e4"
server_key_iv: "de407b54b78ec39df3bac93aa7ad0722"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:55.628660 20893 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 13:10:55.658142 20910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:55.658465 20893 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.000s sys 0.001s
I20260928 13:10:55.658586 20893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "710e390f949e475da4fa7c8eb1a71d25"
format_stamp: "Formatted at 2026-09-28 13:10:55 on dist-test-slave-p1lc"
server_key: "0c08d3edbb087c31569268ee390ff2e4"
server_key_iv: "de407b54b78ec39df3bac93aa7ad0722"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:55.658710 20893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:10:55.679419 20893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:10:55.682425 20893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:10:55.682686 20893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:10:55.683043 20893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:10:55.683475 20893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:10:55.683565 20893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:55.683682 20893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:10:55.683807 20893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:55.691087 20893 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:44187
I20260928 13:10:55.691658 20893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:10:55.693663 21023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:44187 every 8 connection(s)
I20260928 13:10:55.694264 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 20893
I20260928 13:10:55.694366 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:10:55.694584 8546 external_mini_cluster.cc:1428] Setting key 2622f9c79122561b7cb842c41325d8ce
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Sep 28 13:10:55 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:10:55.724408 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:55.697103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46263 (local address 127.8.88.129:58667)
0928 13:10:55.699033 (+ 1930us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:55.699108 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:55.699785 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:55.709387 (+ 9602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:55.709394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:55.709702 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:10:55.710041 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:55.710049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:55.711235 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:55.711237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:55.712595 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:55.712599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:55.713212 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:55.713214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:55.713362 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:55.714010 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:10:55.714021 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:10:55.719988 (+ 5967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:10:55.722496 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:55.722512 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:55.722516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:55.722910 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:55.723162 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:55.723163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:55.723163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:55.723316 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:55.723628 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:10:55.723635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:10:55.723833 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 13:10:55.723955 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 13:10:55.724148 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1875,"thread_start_us":121,"threads_started":1}
I20260928 13:10:55.725294 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:55.694806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:58667 (local address 127.8.88.190:46263)
0928 13:10:55.708983 (+ 14177us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:55.708987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:55.709004 (+ 17us) server_negotiation.cc:407] Connection header received
0928 13:10:55.709039 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:55.709040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:55.709057 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:55.709160 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:55.710225 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:55.711056 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:55.712727 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:55.713048 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:55.713119 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:55.720145 (+ 7026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:55.720147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:55.720148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:55.720157 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:55.722357 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:55.723014 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:55.723016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:55.723016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:55.723089 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:55.723389 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:55.723390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:55.723391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:55.723544 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:55.723595 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:55.725092 (+ 1497us) server_negotiation.cc:299] Negotiation successful
0928 13:10:55.725194 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14142,"thread_start_us":58,"threads_started":1}
I20260928 13:10:55.725677 21024 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46263
I20260928 13:10:55.725775 21024 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:55.726001 21024 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
I20260928 13:10:55.726588 20812 ts_manager.cc:194] Registered new tserver with Master: 710e390f949e475da4fa7c8eb1a71d25 (127.8.88.129:44187)
I20260928 13:10:55.727550 20812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:58667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:10:55.763868 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.88.148:34603/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46263
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 13:10:55 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790601055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 13:10:55.892170 21032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:10:55.892715 21032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:10:55.892812 21032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:10:55.894762 21032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:10:55.895033 21032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:10:55.895136 21032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:10:55.897676 21032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.88.148:34603/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:10:55.898313 21032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:10:55.898672 21032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:10:55.905665 21041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:55.905689 21039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:55.905683 21038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:55.907135 21032 server_base.cc:1034] running on GCE node
I20260928 13:10:55.907367 21032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:10:55.907636 21032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:10:55.908835 21032 hybrid_clock.cc:648] HybridClock initialized: now 1790601055908812 us; error 33 us; skew 500 ppm
Sep 28 13:10:55 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:10:55.912153 21032 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:10:55.912700 21032 webserver.cc:460] Webserver started at http://127.8.88.130:44307/ using document root <none> and password file <none>
I20260928 13:10:55.913128 21032 fs_manager.cc:359] Metadata directory not provided
I20260928 13:10:55.913259 21032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:10:55.913440 21032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 13:10:55 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:55.929791 21032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "e5dacf0a02174d248e2d1b9753c570f0"
format_stamp: "Formatted at 2026-09-28 13:10:55 on dist-test-slave-p1lc"
server_key: "00849aeaabb1b0d3eb76d13c3885f0f1"
server_key_iv: "77006613f1306e100dcff467290791bb"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:55.930290 21032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "e5dacf0a02174d248e2d1b9753c570f0"
format_stamp: "Formatted at 2026-09-28 13:10:55 on dist-test-slave-p1lc"
server_key: "00849aeaabb1b0d3eb76d13c3885f0f1"
server_key_iv: "77006613f1306e100dcff467290791bb"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:55.937419 21032 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.000s
I20260928 13:10:55.960292 21048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:55.960489 21032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 13:10:55.960579 21032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "e5dacf0a02174d248e2d1b9753c570f0"
format_stamp: "Formatted at 2026-09-28 13:10:55 on dist-test-slave-p1lc"
server_key: "00849aeaabb1b0d3eb76d13c3885f0f1"
server_key_iv: "77006613f1306e100dcff467290791bb"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:55.960688 21032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:10:55.979209 21032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:10:55.981686 21032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:10:55.981892 21032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:10:55.982159 21032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:10:55.982545 21032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:10:55.982628 21032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:55.982697 21032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:10:55.982753 21032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:55.991312 21032 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:32881
I20260928 13:10:55.991883 21032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:10:55.992570 21161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:32881 every 8 connection(s)
I20260928 13:10:55.994118 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 21032
I20260928 13:10:55.994230 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:10:55.994446 8546 external_mini_cluster.cc:1428] Setting key 2aaeb0c0819b9af9c15cfb1612afdadb
Sep 28 13:10:55 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:10:56.004432 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:55.993678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:44097 (local address 127.8.88.190:46263)
0928 13:10:55.993743 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:55.993745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:55.993902 (+ 157us) server_negotiation.cc:407] Connection header received
0928 13:10:55.994784 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:55.994786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:55.994803 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:55.994833 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:55.995691 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:55.996447 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:55.998004 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:55.998317 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:55.998363 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:56.001055 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:56.001057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:56.001057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:56.001065 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:56.002827 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:56.003354 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:56.003356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:56.003356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:56.003422 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:56.003690 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:56.003691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:56.003691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:56.003827 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:56.003866 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:56.004243 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 13:10:56.004335 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 13:10:56.005256 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:55.993394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46263 (local address 127.8.88.130:44097)
0928 13:10:55.993773 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:55.993821 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:55.994663 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:55.995028 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:55.995033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:55.995286 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:10:55.995581 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:55.995588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:55.996565 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:55.996566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:55.997890 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:55.997893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:55.998473 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:55.998474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:55.998619 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:55.999255 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:10:55.999267 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:10:56.000946 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:10:56.002954 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:56.002961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:56.002964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:56.003268 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:56.003491 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:56.003491 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:56.003492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:56.003621 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:56.003908 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:10:56.003914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:10:56.004079 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 13:10:56.004805 (+ 726us) client_negotiation.cc:240] Negotiation successful
0928 13:10:56.005002 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":80,"threads_started":1}
I20260928 13:10:56.007555 21162 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46263
I20260928 13:10:56.007658 21162 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:56.007851 21162 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
I20260928 13:10:56.008294 20812 ts_manager.cc:194] Registered new tserver with Master: e5dacf0a02174d248e2d1b9753c570f0 (127.8.88.130:32881)
I20260928 13:10:56.008796 20812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:44097
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:10:56.049358 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.88.148:34603/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46263
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:10:56.182861 21167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:10:56.183239 21167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:10:56.183332 21167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:10:56.185177 21167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:10:56.185425 21167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:10:56.185511 21167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:10:56.187788 21167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.88.148:34603/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:10:56.188376 21167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:10:56.188721 21167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:10:56.195807 21173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:56.196187 21167 server_base.cc:1034] running on GCE node
W20260928 13:10:56.196436 21172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:56.196663 21175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:56.196972 21167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:10:56.197255 21167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:10:56.198400 21167 hybrid_clock.cc:648] HybridClock initialized: now 1790601056198374 us; error 39 us; skew 500 ppm
Sep 28 13:10:56 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:10:56.202749 21167 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:10:56.203513 21167 webserver.cc:460] Webserver started at http://127.8.88.131:38755/ using document root <none> and password file <none>
I20260928 13:10:56.203948 21167 fs_manager.cc:359] Metadata directory not provided
I20260928 13:10:56.204083 21167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:10:56.204299 21167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 13:10:56 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:56.223601 21167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "faaa573976c645998b868a2f02157543"
format_stamp: "Formatted at 2026-09-28 13:10:56 on dist-test-slave-p1lc"
server_key: "bf8e96fe3f167ec92dfd3e0a1908579c"
server_key_iv: "d29c7a84f2ad09593114fb18ea67637c"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:56.224074 21167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "faaa573976c645998b868a2f02157543"
format_stamp: "Formatted at 2026-09-28 13:10:56 on dist-test-slave-p1lc"
server_key: "bf8e96fe3f167ec92dfd3e0a1908579c"
server_key_iv: "d29c7a84f2ad09593114fb18ea67637c"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:56.238590 21167 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.000s
I20260928 13:10:56.253968 21182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:56.254141 21167 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:10:56.254227 21167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "faaa573976c645998b868a2f02157543"
format_stamp: "Formatted at 2026-09-28 13:10:56 on dist-test-slave-p1lc"
server_key: "bf8e96fe3f167ec92dfd3e0a1908579c"
server_key_iv: "d29c7a84f2ad09593114fb18ea67637c"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:56.254333 21167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:10:56.267473 21167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:10:56.271884 21167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:10:56.273692 21167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:10:56.274008 21167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:10:56.274384 21167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:10:56.274430 21167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:56.274469 21167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:10:56.274487 21167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:56.286458 21167 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:35693
I20260928 13:10:56.286520 21295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:35693 every 8 connection(s)
I20260928 13:10:56.286899 21167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
Sep 28 13:10:56 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:10:56.294934 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 21167
I20260928 13:10:56.295032 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:10:56.295264 8546 external_mini_cluster.cc:1428] Setting key 95a4bcd4153c54e307d7142033227db6
I20260928 13:10:56.297326 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:56.287761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:58087 (local address 127.8.88.190:46263)
0928 13:10:56.287837 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:56.287840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:56.288056 (+ 216us) server_negotiation.cc:407] Connection header received
0928 13:10:56.288626 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:56.288628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:56.288642 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:56.288709 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:56.289408 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:56.289904 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:56.290945 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:56.291144 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:56.291234 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:56.293878 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:56.293879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:56.293880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:56.293889 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:56.295728 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:56.296269 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:56.296270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:56.296271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:56.296330 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:56.296591 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:56.296591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:56.296592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:56.296728 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:56.296800 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:56.297143 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 13:10:56.297238 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 13:10:56.297559 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:56.287733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46263 (local address 127.8.88.131:58087)
0928 13:10:56.287921 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:56.287962 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:56.288509 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:56.288798 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:56.288801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:56.289075 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:10:56.289292 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:56.289297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:56.290004 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:56.290006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:56.290849 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:56.290851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:56.291254 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:56.291255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:56.291376 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:56.292011 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:10:56.292025 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:10:56.293744 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:10:56.295838 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:56.295846 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:56.295849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:56.296206 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:56.296407 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:56.296408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:56.296409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:56.296516 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:56.296802 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:10:56.296809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:10:56.297010 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 13:10:56.297124 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 13:10:56.297276 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260928 13:10:56.298076 21296 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46263
I20260928 13:10:56.298220 21296 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:56.298420 21296 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
I20260928 13:10:56.298833 20812 ts_manager.cc:194] Registered new tserver with Master: faaa573976c645998b868a2f02157543 (127.8.88.131:35693)
I20260928 13:10:56.299230 20812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:58087
I20260928 13:10:56.307096 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:10:56.307806 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 20893
I20260928 13:10:56.314332 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 21032
I20260928 13:10:56.319416 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 21167
I20260928 13:10:56.327603 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 20780
I20260928 13:10:56.339525 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46263
--webserver_interface=127.8.88.190
--webserver_port=46387
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46263
--ranger_config_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.88.148:34603/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.88.148:34603/kms with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:10:56.429363 21299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:10:56.429579 21299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:10:56.429601 21299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:10:56.430863 21299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:10:56.430917 21299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:10:56.430930 21299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:10:56.430943 21299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:10:56.430955 21299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:10:56.432597 21299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.88.148:34603/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46263
--ranger_config_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=46387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:10:56.432865 21299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:10:56.433156 21299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:10:56.438350 21299 server_base.cc:1034] running on GCE node
W20260928 13:10:56.438335 21304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:56.438338 21305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:56.438557 21307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:56.438833 21299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:10:56.439102 21299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:10:56.440275 21299 hybrid_clock.cc:648] HybridClock initialized: now 1790601056440257 us; error 38 us; skew 500 ppm
Sep 28 13:10:56 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:10:56.443089 21299 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:10:56.443536 21299 webserver.cc:460] Webserver started at http://127.8.88.190:46387/ using document root <none> and password file <none>
I20260928 13:10:56.443778 21299 fs_manager.cc:359] Metadata directory not provided
I20260928 13:10:56.443856 21299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 13:10:56 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:56.490095 21299 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:10:56.490976 21315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:56.491231 21299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:10:56.491350 21299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "01017f7c9e5f4ed48e150b049ce0c458"
format_stamp: "Formatted at 2026-09-28 13:10:54 on dist-test-slave-p1lc"
server_key: "fc65ea7a56946ee231ee0b8a5b6217b1"
server_key_iv: "20c391d89fbe7ce34c11dbc52275acb2"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:56.491776 21299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:10:56.501772 21299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:10:56.503693 21299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:10:56.503859 21299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:10:56.508229 21299 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:46263
I20260928 13:10:56.508266 21367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:46263 every 8 connection(s)
I20260928 13:10:56.508641 21299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:10:56.510035 21368 sys_catalog.cc:263] Verifying existing consensus state
I20260928 13:10:56.510730 21368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458: Bootstrap starting.
I20260928 13:10:56.515986 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 21299
I20260928 13:10:56.516309 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.88.148:34603/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:44187
--local_ip_for_outbound_sockets=127.8.88.129
--tserver_master_addrs=127.8.88.190:46263
--webserver_port=35665
--webserver_interface=127.8.88.129
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.88.148:34603/kms with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:10:56.519085 21368 log.cc:824] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458: Log is configured to *not* fsync() on all Append() calls
I20260928 13:10:56.521270 21368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:10:56.521574 21368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458: Bootstrap complete.
I20260928 13:10:56.523943 21368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01017f7c9e5f4ed48e150b049ce0c458" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46263 } }
I20260928 13:10:56.524190 21368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01017f7c9e5f4ed48e150b049ce0c458, State: Initialized, Role: FOLLOWER
I20260928 13:10:56.524364 21368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01017f7c9e5f4ed48e150b049ce0c458" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46263 } }
I20260928 13:10:56.524433 21368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:10:56.524478 21368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:10:56.524555 21368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 1 FOLLOWER]: Advancing to term 2
I20260928 13:10:56.525687 21368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01017f7c9e5f4ed48e150b049ce0c458" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46263 } }
I20260928 13:10:56.525780 21368 leader_election.cc:302] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01017f7c9e5f4ed48e150b049ce0c458; no voters:
I20260928 13:10:56.525991 21368 leader_election.cc:288] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 13:10:56.526031 21372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 13:10:56.526175 21372 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [term 2 LEADER]: Becoming Leader. State: Replica: 01017f7c9e5f4ed48e150b049ce0c458, State: Running, Role: LEADER
I20260928 13:10:56.526288 21368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:10:56.526341 21372 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01017f7c9e5f4ed48e150b049ce0c458" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46263 } }
I20260928 13:10:56.526640 21372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01017f7c9e5f4ed48e150b049ce0c458. Latest consensus state: current_term: 2 leader_uuid: "01017f7c9e5f4ed48e150b049ce0c458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01017f7c9e5f4ed48e150b049ce0c458" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46263 } } }
I20260928 13:10:56.526769 21372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [sys.catalog]: This master's current role is: LEADER
I20260928 13:10:56.526645 21373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "01017f7c9e5f4ed48e150b049ce0c458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01017f7c9e5f4ed48e150b049ce0c458" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46263 } } }
I20260928 13:10:56.526963 21373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458 [sys.catalog]: This master's current role is: LEADER
I20260928 13:10:56.527063 21381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:10:56.527611 21381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:10:56.527797 21381 catalog_manager.cc:1259] Loaded cluster ID: 80bb545777eb416892d7af6e18caa6be
I20260928 13:10:56.527869 21381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:10:56.527993 21368 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 13:10:56.529141 21381 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:10:56.529325 21381 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458: Loaded TSK: 0
I20260928 13:10:56.529543 21381 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 13:10:56.632642 21370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:10:56.633082 21370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:10:56.633162 21370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:10:56.635286 21370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:10:56.635406 21370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:10:56.635484 21370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:10:56.637928 21370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.88.148:34603/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:44187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=35665
--tserver_master_addrs=127.8.88.190:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:10:56.638236 21370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:10:56.638562 21370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:10:56.646117 21408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:56.646113 21406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:56.647606 21370 server_base.cc:1034] running on GCE node
W20260928 13:10:56.649060 21405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:56.649415 21370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:10:56.649770 21370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:10:56.650954 21370 hybrid_clock.cc:648] HybridClock initialized: now 1790601056650901 us; error 70 us; skew 500 ppm
Sep 28 13:10:56 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:10:56.654583 21370 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:10:56.655153 21370 webserver.cc:460] Webserver started at http://127.8.88.129:35665/ using document root <none> and password file <none>
I20260928 13:10:56.655441 21370 fs_manager.cc:359] Metadata directory not provided
I20260928 13:10:56.655535 21370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 13:10:56 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:56.685577 21370 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.001s sys 0.000s
I20260928 13:10:56.693224 21416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:56.693517 21370 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260928 13:10:56.693645 21370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "710e390f949e475da4fa7c8eb1a71d25"
format_stamp: "Formatted at 2026-09-28 13:10:55 on dist-test-slave-p1lc"
server_key: "0c08d3edbb087c31569268ee390ff2e4"
server_key_iv: "de407b54b78ec39df3bac93aa7ad0722"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:56.694209 21370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:10:56.720537 21370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:10:56.725405 21370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:10:56.725630 21370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:10:56.725906 21370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:10:56.726310 21370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:10:56.726392 21370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:56.726451 21370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:10:56.726483 21370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:56.733757 21370 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:44187
I20260928 13:10:56.733945 21529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:44187 every 8 connection(s)
I20260928 13:10:56.734315 21370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:10:56.743522 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 21370
I20260928 13:10:56.743862 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.88.148:34603/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:32881
--local_ip_for_outbound_sockets=127.8.88.130
--tserver_master_addrs=127.8.88.190:46263
--webserver_port=44307
--webserver_interface=127.8.88.130
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.88.148:34603/kms with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 13:10:56 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:10:56.756057 21532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:56.735144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46263 (local address 127.8.88.129:34589)
0928 13:10:56.735366 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:56.735416 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:56.736026 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:56.736625 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:56.736631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:56.736918 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:10:56.737235 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:56.737243 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:56.741761 (+ 4518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:56.741763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:56.743223 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:56.743227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:56.744305 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:56.744307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:56.744444 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:56.749681 (+ 5237us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:10:56.749693 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:10:56.751436 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:10:56.753948 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:56.753957 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:56.753959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:56.754311 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:56.754636 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:56.754637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:56.754637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:56.754779 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:56.755345 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:10:56.755355 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:10:56.755518 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 13:10:56.755635 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 13:10:56.755809 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260928 13:10:56.756690 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:56.735573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:34589 (local address 127.8.88.190:46263)
0928 13:10:56.735814 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:56.735825 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:56.735865 (+ 40us) server_negotiation.cc:407] Connection header received
0928 13:10:56.736311 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:56.736320 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:56.736391 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:56.736556 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:56.737355 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:56.741593 (+ 4238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:56.743345 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:56.744044 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:56.744146 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:56.751606 (+ 7460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:56.751612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:56.751613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:56.751627 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:56.753835 (+ 2208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:56.754423 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:56.754427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:56.754427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:56.754552 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:56.754859 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:56.754860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:56.754861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:56.755207 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:56.755294 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:56.756334 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0928 13:10:56.756476 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260928 13:10:56.758297 21530 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46263
I20260928 13:10:56.758412 21530 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:56.758950 21530 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
W20260928 13:10:56.760241 21530 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:46263 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 13:10:56.760313 21530 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:56.760378 21530 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
I20260928 13:10:56.760787 21530 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:56.760895 21530 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
W20260928 13:10:56.761252 21530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 13:10:56.886885 21534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:10:56.887238 21534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:10:56.887292 21534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:10:56.889459 21534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:10:56.889534 21534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:10:56.889581 21534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:10:56.892190 21534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.88.148:34603/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:32881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=44307
--tserver_master_addrs=127.8.88.190:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:10:56.892489 21534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:10:56.892810 21534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:10:56.899156 21540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:56.899192 21539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:56.899194 21542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:56.899884 21534 server_base.cc:1034] running on GCE node
I20260928 13:10:56.900158 21534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:10:56.900434 21534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:10:56.907838 21534 hybrid_clock.cc:648] HybridClock initialized: now 1790601056907798 us; error 53 us; skew 500 ppm
Sep 28 13:10:56 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:10:56.910835 21534 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:10:56.911302 21534 webserver.cc:460] Webserver started at http://127.8.88.130:44307/ using document root <none> and password file <none>
I20260928 13:10:56.911549 21534 fs_manager.cc:359] Metadata directory not provided
I20260928 13:10:56.911633 21534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 13:10:56 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:56.926860 21534 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:10:56.927927 21550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:56.928087 21534 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:10:56.928158 21534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "e5dacf0a02174d248e2d1b9753c570f0"
format_stamp: "Formatted at 2026-09-28 13:10:55 on dist-test-slave-p1lc"
server_key: "00849aeaabb1b0d3eb76d13c3885f0f1"
server_key_iv: "77006613f1306e100dcff467290791bb"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:56.928615 21534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:10:56.953240 21534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:10:56.955947 21534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:10:56.956141 21534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:10:56.956413 21534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:10:56.956825 21534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:10:56.956904 21534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:56.956943 21534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:10:56.956962 21534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:56.965286 21534 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:32881
I20260928 13:10:56.965390 21663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:32881 every 8 connection(s)
I20260928 13:10:56.965842 21534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:10:56.967360 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 21534
I20260928 13:10:56.967701 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.88.148:34603/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:35693
--local_ip_for_outbound_sockets=127.8.88.131
--tserver_master_addrs=127.8.88.190:46263
--webserver_port=38755
--webserver_interface=127.8.88.131
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.88.148:34603/kms with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 13:10:56 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:10:56.978173 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:56.967088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:37143 (local address 127.8.88.190:46263)
0928 13:10:56.967204 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:56.967207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:56.967217 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:10:56.968071 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:56.968073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:56.968091 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:56.968177 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:56.968805 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:56.969530 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:56.971167 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:56.971514 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:56.971559 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:56.974465 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:56.974467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:56.974467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:56.974477 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:56.976349 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:56.976972 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:56.976973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:56.976974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:56.977039 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:56.977354 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:56.977355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:56.977355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:56.977500 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:56.977546 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:56.977970 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 13:10:56.978070 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 13:10:56.979017 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:56.966738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46263 (local address 127.8.88.130:37143)
0928 13:10:56.966968 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:56.967019 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:56.967879 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:56.968263 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:56.968272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:56.968470 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:10:56.968687 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:56.968692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:56.969667 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:56.969668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:56.971053 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:56.971056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:56.971691 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:56.971692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:56.971855 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:56.972692 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:10:56.972705 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:10:56.974343 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:10:56.976464 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:56.976473 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:56.976476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:56.976829 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:56.977116 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:56.977116 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:56.977117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:56.977270 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:56.977596 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:10:56.977603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:10:56.977803 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 13:10:56.978561 (+ 758us) client_negotiation.cc:240] Negotiation successful
0928 13:10:56.978750 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":109,"threads_started":1}
I20260928 13:10:56.979739 21664 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46263
I20260928 13:10:56.979849 21664 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:56.980044 21664 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
W20260928 13:10:56.980720 21664 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:46263 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 13:10:56.980803 21664 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:56.980886 21664 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
I20260928 13:10:56.981235 21664 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:56.981307 21664 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
W20260928 13:10:56.981630 21664 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 13:10:57.081519 21667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:10:57.081941 21667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:10:57.082033 21667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:10:57.084015 21667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:10:57.084138 21667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:10:57.084252 21667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:10:57.086736 21667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.88.148:34603/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:35693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=38755
--tserver_master_addrs=127.8.88.190:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:10:57.087333 21667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:10:57.087673 21667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:10:57.093693 21676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:57.093693 21673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:10:57.094188 21674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:10:57.095044 21667 server_base.cc:1034] running on GCE node
I20260928 13:10:57.095325 21667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:10:57.095634 21667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:10:57.096836 21667 hybrid_clock.cc:648] HybridClock initialized: now 1790601057096820 us; error 27 us; skew 500 ppm
Sep 28 13:10:57 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601057, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:10:57.100591 21667 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:10:57.101263 21667 webserver.cc:460] Webserver started at http://127.8.88.131:38755/ using document root <none> and password file <none>
I20260928 13:10:57.101563 21667 fs_manager.cc:359] Metadata directory not provided
I20260928 13:10:57.101652 21667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 13:10:57 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601057, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:57.121404 21667 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 13:10:57.122589 21684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:57.122980 21667 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:10:57.123255 21667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "faaa573976c645998b868a2f02157543"
format_stamp: "Formatted at 2026-09-28 13:10:56 on dist-test-slave-p1lc"
server_key: "bf8e96fe3f167ec92dfd3e0a1908579c"
server_key_iv: "d29c7a84f2ad09593114fb18ea67637c"
server_key_version: "kuduclusterkey@0"
I20260928 13:10:57.123824 21667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:10:57.133441 21667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:10:57.135874 21667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:10:57.136051 21667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:10:57.136337 21667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:10:57.136752 21667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:10:57.136806 21667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:57.136838 21667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:10:57.136901 21667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:10:57.147315 21667 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:35693
I20260928 13:10:57.147406 21797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:35693 every 8 connection(s)
I20260928 13:10:57.147766 21667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:10:57.155107 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 21667
Sep 28 13:10:57 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601057, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:10:57.160846 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:57.148612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:55063 (local address 127.8.88.190:46263)
0928 13:10:57.148684 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:57.148687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:57.148994 (+ 307us) server_negotiation.cc:407] Connection header received
0928 13:10:57.149724 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:57.149726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:57.149739 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:57.149771 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:57.150655 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:57.151458 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:57.153198 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:57.153556 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:57.153662 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:57.156727 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:57.156728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:57.156729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:57.156737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:57.158685 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:57.159510 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:57.159511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:57.159512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:57.159583 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:57.159979 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:57.159980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:57.159980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:57.160110 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:57.160158 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:57.160595 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 13:10:57.160721 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 13:10:57.161547 21800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:57.148623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46263 (local address 127.8.88.131:55063)
0928 13:10:57.148817 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:57.148855 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:57.149577 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:57.149983 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:57.149987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:57.150230 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:10:57.150517 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:57.150524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:57.151593 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:57.151594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:57.153082 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:57.153086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:57.153715 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:57.153717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:57.153872 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:57.154537 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:10:57.154548 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:10:57.156569 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:10:57.158773 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:57.158784 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:57.158787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:57.159409 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:57.159680 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:10:57.159681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:10:57.159681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:10:57.159895 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:10:57.160214 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:10:57.160221 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:10:57.160411 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 13:10:57.161031 (+ 620us) client_negotiation.cc:240] Negotiation successful
0928 13:10:57.161275 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":101,"threads_started":1}
I20260928 13:10:57.162046 21798 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46263
I20260928 13:10:57.162163 21798 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:57.162384 21798 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
W20260928 13:10:57.163075 21798 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:46263 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 13:10:57.163154 21798 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:57.163230 21798 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
I20260928 13:10:57.163548 21798 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:57.163627 21798 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
W20260928 13:10:57.163923 21798 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 13:10:57.164608 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:57.155562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.8.88.190:46263)
0928 13:10:57.155940 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:57.155942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:57.155950 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:10:57.155974 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:57.155975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:57.155988 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:57.156065 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:57.156948 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:57.158403 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:57.159608 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:57.159903 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:57.159964 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:57.161554 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:57.161556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:57.161556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:57.161563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:57.163350 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:57.163762 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:57.163763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:57.163763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:57.163826 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:57.164043 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:57.164044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:57.164044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:57.164169 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:57.164359 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:57.164400 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 13:10:57.164487 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":98,"threads_started":1}
Sep 28 13:10:57 dist-test-slave-p1lc krb5kdc[20336](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790601057, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:10:57.761895 21530 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:57.762065 21530 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
I20260928 13:10:57.762559 21332 ts_manager.cc:194] Registered new tserver with Master: 710e390f949e475da4fa7c8eb1a71d25 (127.8.88.129:44187)
I20260928 13:10:57.763273 21332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:34589
I20260928 13:10:57.984970 21664 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:57.985145 21664 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
I20260928 13:10:57.985405 21332 ts_manager.cc:194] Registered new tserver with Master: e5dacf0a02174d248e2d1b9753c570f0 (127.8.88.130:32881)
I20260928 13:10:57.985955 21332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:37143
Sep 28 13:10:58 dist-test-slave-p1lc krb5kdc[20336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601057, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for HTTP/127.8.88.148@KRBTEST.COM
I20260928 13:10:58.164111 21798 heartbeater.cc:464] Registering TS with master...
I20260928 13:10:58.164292 21798 heartbeater.cc:511] Master 127.8.88.190:46263 requested a full tablet report, sending...
I20260928 13:10:58.164688 21332 ts_manager.cc:194] Registered new tserver with Master: faaa573976c645998b868a2f02157543 (127.8.88.131:35693)
I20260928 13:10:58.165220 21332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:55063
I20260928 13:10:58.172181 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:10:58.181150 21820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.173295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.8.88.190:46263)
0928 13:10:58.173451 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:58.173453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:58.173590 (+ 137us) server_negotiation.cc:407] Connection header received
0928 13:10:58.173683 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:58.173684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:58.173696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:58.173737 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:10:58.174505 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.175278 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.176322 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.176608 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.176649 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:58.178168 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:10:58.178170 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:10:58.178170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:10:58.178178 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:10:58.179897 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:58.180300 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:58.180301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:58.180301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:58.180356 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:10:58.180536 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:10:58.180537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:10:58.180537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:10:58.180664 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:10:58.180734 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:58.180966 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 13:10:58.181050 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":57,"threads_started":1}
I20260928 13:10:58.183910 21332 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:10:58.186113 21332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:10:58.201941 21828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.189199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:44187 (local address 127.0.0.1:36750)
0928 13:10:58.197072 (+ 7873us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:58.197132 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:58.197230 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:58.197543 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:58.197546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:58.197549 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:10:58.197629 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:58.197630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.198871 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.198871 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:58.200153 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:58.200156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.201540 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.201541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:58.201618 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:58.201632 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 13:10:58.201707 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 13:10:58.201759 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7834,"thread_start_us":65,"threads_started":1}
I20260928 13:10:58.202394 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.189388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:35693 (local address 127.0.0.1:54220)
0928 13:10:58.197465 (+ 8077us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:58.197476 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:58.197528 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:58.197808 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:58.197809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:58.197811 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:10:58.197854 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:58.197856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.199089 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.199090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:58.200219 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:58.200222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.202157 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.202159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:58.202218 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:58.202222 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:10:58.202277 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 13:10:58.202317 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8059,"thread_start_us":61,"threads_started":1}
I20260928 13:10:58.202471 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.189291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.8.88.129:44187)
0928 13:10:58.196976 (+ 7685us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:58.196984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:58.197231 (+ 247us) server_negotiation.cc:407] Connection header received
0928 13:10:58.197322 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:58.197325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:58.197343 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:58.197422 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:10:58.197720 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.198780 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.200527 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.201385 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.201451 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:58.201506 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:58.202357 (+ 851us) server_negotiation.cc:299] Negotiation successful
0928 13:10:58.202399 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7637,"thread_start_us":63,"threads_started":1}
I20260928 13:10:58.203168 21464 tablet_service.cc:1467] Processing CreateTablet for tablet e1e0a0cad16f4d538d9adea0ee59a92e (DEFAULT_TABLE table=test-table [id=6c32f4af95c440f9864772f258f39587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:10:58.203558 21464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e0a0cad16f4d538d9adea0ee59a92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:10:58.207252 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.189513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.8.88.131:35693)
0928 13:10:58.197395 (+ 7882us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:58.197400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:58.197584 (+ 184us) server_negotiation.cc:407] Connection header received
0928 13:10:58.197624 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:58.197627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:58.197653 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:58.197719 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:10:58.197926 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.199007 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.200321 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.201200 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.201263 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:58.201318 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:58.207111 (+ 5793us) server_negotiation.cc:299] Negotiation successful
0928 13:10:58.207171 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7842,"thread_start_us":64,"threads_started":1}
I20260928 13:10:58.207487 21831 tablet_bootstrap.cc:492] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25: Bootstrap starting.
I20260928 13:10:58.208179 21831 tablet_bootstrap.cc:654] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25: Neither blocks nor log segments found. Creating new log.
I20260928 13:10:58.208081 21732 tablet_service.cc:1467] Processing CreateTablet for tablet e1e0a0cad16f4d538d9adea0ee59a92e (DEFAULT_TABLE table=test-table [id=6c32f4af95c440f9864772f258f39587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:10:58.208447 21732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e0a0cad16f4d538d9adea0ee59a92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:10:58.208598 21831 log.cc:824] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25: Log is configured to *not* fsync() on all Append() calls
I20260928 13:10:58.211757 21833 tablet_bootstrap.cc:492] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543: Bootstrap starting.
I20260928 13:10:58.212374 21833 tablet_bootstrap.cc:654] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543: Neither blocks nor log segments found. Creating new log.
I20260928 13:10:58.212747 21833 log.cc:824] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543: Log is configured to *not* fsync() on all Append() calls
I20260928 13:10:58.217206 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.189682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.8.88.130:32881)
0928 13:10:58.198709 (+ 9027us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:58.198714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:58.198723 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:10:58.198756 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:58.198758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:58.198771 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:58.198889 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:10:58.199087 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.200250 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.201484 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.207917 (+ 6433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.207998 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:58.208063 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:58.217037 (+ 8974us) server_negotiation.cc:299] Negotiation successful
0928 13:10:58.217121 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8992,"thread_start_us":60,"threads_started":1}
I20260928 13:10:58.217233 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.189516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:32881 (local address 127.0.0.1:41746)
0928 13:10:58.198535 (+ 9019us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:58.198546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:58.198597 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:58.198963 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:58.198963 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:58.198965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:10:58.199008 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:58.199010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.200331 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.200332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:58.201396 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:58.201399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.216771 (+ 15372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.216772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:58.216912 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:58.216921 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 13:10:58.216993 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 13:10:58.217054 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9001,"thread_start_us":50,"threads_started":1}
I20260928 13:10:58.218201 21598 tablet_service.cc:1467] Processing CreateTablet for tablet e1e0a0cad16f4d538d9adea0ee59a92e (DEFAULT_TABLE table=test-table [id=6c32f4af95c440f9864772f258f39587]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:10:58.218333 21831 tablet_bootstrap.cc:492] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25: No bootstrap required, opened a new log
I20260928 13:10:58.218410 21831 ts_tablet_manager.cc:1397] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260928 13:10:58.218568 21598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e0a0cad16f4d538d9adea0ee59a92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:10:58.220628 21831 raft_consensus.cc:348] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710e390f949e475da4fa7c8eb1a71d25" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44187 } } peers { permanent_uuid: "faaa573976c645998b868a2f02157543" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35693 } } peers { permanent_uuid: "e5dacf0a02174d248e2d1b9753c570f0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 32881 } }
I20260928 13:10:58.220814 21831 raft_consensus.cc:374] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:10:58.220849 21831 raft_consensus.cc:729] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710e390f949e475da4fa7c8eb1a71d25, State: Initialized, Role: FOLLOWER
I20260928 13:10:58.221007 21831 consensus_queue.cc:260] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710e390f949e475da4fa7c8eb1a71d25" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44187 } } peers { permanent_uuid: "faaa573976c645998b868a2f02157543" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35693 } } peers { permanent_uuid: "e5dacf0a02174d248e2d1b9753c570f0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 32881 } }
I20260928 13:10:58.221302 21831 ts_tablet_manager.cc:1428] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:10:58.221423 21530 heartbeater.cc:503] Master 127.8.88.190:46263 was elected leader, sending a full tablet report...
I20260928 13:10:58.222164 21833 tablet_bootstrap.cc:492] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543: No bootstrap required, opened a new log
I20260928 13:10:58.222239 21833 ts_tablet_manager.cc:1397] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260928 13:10:58.222790 21836 tablet_bootstrap.cc:492] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0: Bootstrap starting.
I20260928 13:10:58.223474 21836 tablet_bootstrap.cc:654] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0: Neither blocks nor log segments found. Creating new log.
I20260928 13:10:58.223857 21836 log.cc:824] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0: Log is configured to *not* fsync() on all Append() calls
I20260928 13:10:58.224728 21836 tablet_bootstrap.cc:492] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0: No bootstrap required, opened a new log
I20260928 13:10:58.224792 21836 ts_tablet_manager.cc:1397] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:10:58.225111 21833 raft_consensus.cc:348] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710e390f949e475da4fa7c8eb1a71d25" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44187 } } peers { permanent_uuid: "faaa573976c645998b868a2f02157543" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35693 } } peers { permanent_uuid: "e5dacf0a02174d248e2d1b9753c570f0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 32881 } }
I20260928 13:10:58.225275 21833 raft_consensus.cc:374] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:10:58.225309 21833 raft_consensus.cc:729] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faaa573976c645998b868a2f02157543, State: Initialized, Role: FOLLOWER
I20260928 13:10:58.225436 21833 consensus_queue.cc:260] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710e390f949e475da4fa7c8eb1a71d25" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44187 } } peers { permanent_uuid: "faaa573976c645998b868a2f02157543" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35693 } } peers { permanent_uuid: "e5dacf0a02174d248e2d1b9753c570f0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 32881 } }
I20260928 13:10:58.225711 21833 ts_tablet_manager.cc:1428] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 13:10:58.225827 21798 heartbeater.cc:503] Master 127.8.88.190:46263 was elected leader, sending a full tablet report...
I20260928 13:10:58.227165 21836 raft_consensus.cc:348] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710e390f949e475da4fa7c8eb1a71d25" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44187 } } peers { permanent_uuid: "faaa573976c645998b868a2f02157543" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35693 } } peers { permanent_uuid: "e5dacf0a02174d248e2d1b9753c570f0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 32881 } }
I20260928 13:10:58.227345 21836 raft_consensus.cc:374] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:10:58.227378 21836 raft_consensus.cc:729] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5dacf0a02174d248e2d1b9753c570f0, State: Initialized, Role: FOLLOWER
I20260928 13:10:58.227486 21836 consensus_queue.cc:260] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710e390f949e475da4fa7c8eb1a71d25" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44187 } } peers { permanent_uuid: "faaa573976c645998b868a2f02157543" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35693 } } peers { permanent_uuid: "e5dacf0a02174d248e2d1b9753c570f0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 32881 } }
I20260928 13:10:58.227759 21836 ts_tablet_manager.cc:1428] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 13:10:58.227903 21664 heartbeater.cc:503] Master 127.8.88.190:46263 was elected leader, sending a full tablet report...
W20260928 13:10:58.237041 21531 tablet.cc:2369] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:10:58.319975 21838 raft_consensus.cc:482] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:10:58.320135 21838 raft_consensus.cc:504] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710e390f949e475da4fa7c8eb1a71d25" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44187 } } peers { permanent_uuid: "faaa573976c645998b868a2f02157543" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35693 } } peers { permanent_uuid: "e5dacf0a02174d248e2d1b9753c570f0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 32881 } }
I20260928 13:10:58.320526 21838 leader_election.cc:288] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 710e390f949e475da4fa7c8eb1a71d25 (127.8.88.129:44187), e5dacf0a02174d248e2d1b9753c570f0 (127.8.88.130:32881)
I20260928 13:10:58.325099 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.320995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:46761 (local address 127.8.88.129:44187)
0928 13:10:58.321044 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:58.321046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:58.321531 (+ 485us) server_negotiation.cc:407] Connection header received
0928 13:10:58.321635 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:58.321637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:58.321649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:58.321677 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:10:58.321906 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.322635 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.324083 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.324666 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.324717 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:58.324726 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:58.324976 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 13:10:58.325021 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:10:58.325383 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.320776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:44187 (local address 127.8.88.131:46761)
0928 13:10:58.321467 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:58.321477 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:58.321561 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:58.321756 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:58.321757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:58.321759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:10:58.321835 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:58.321837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.322745 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.322745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:58.323968 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:58.323971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.324801 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.324802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:58.324895 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:58.324899 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:10:58.325244 (+ 345us) client_negotiation.cc:240] Negotiation successful
0928 13:10:58.325291 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":307,"threads_started":1}
I20260928 13:10:58.325737 21484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e0a0cad16f4d538d9adea0ee59a92e" candidate_uuid: "faaa573976c645998b868a2f02157543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "710e390f949e475da4fa7c8eb1a71d25" is_pre_election: true
I20260928 13:10:58.325932 21484 raft_consensus.cc:2454] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faaa573976c645998b868a2f02157543 in term 0.
I20260928 13:10:58.326982 21688 leader_election.cc:302] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710e390f949e475da4fa7c8eb1a71d25, faaa573976c645998b868a2f02157543; no voters:
I20260928 13:10:58.327157 21838 raft_consensus.cc:2781] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:10:58.327219 21838 raft_consensus.cc:482] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:10:58.327241 21838 raft_consensus.cc:3037] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:10:58.328181 21838 raft_consensus.cc:504] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710e390f949e475da4fa7c8eb1a71d25" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44187 } } peers { permanent_uuid: "faaa573976c645998b868a2f02157543" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35693 } } peers { permanent_uuid: "e5dacf0a02174d248e2d1b9753c570f0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 32881 } }
I20260928 13:10:58.328320 21838 leader_election.cc:288] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [CANDIDATE]: Term 1 election: Requested vote from peers 710e390f949e475da4fa7c8eb1a71d25 (127.8.88.129:44187), e5dacf0a02174d248e2d1b9753c570f0 (127.8.88.130:32881)
I20260928 13:10:58.328513 21484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e0a0cad16f4d538d9adea0ee59a92e" candidate_uuid: "faaa573976c645998b868a2f02157543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "710e390f949e475da4fa7c8eb1a71d25"
I20260928 13:10:58.328595 21484 raft_consensus.cc:3037] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:10:58.329501 21484 raft_consensus.cc:2454] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faaa573976c645998b868a2f02157543 in term 1.
I20260928 13:10:58.329685 21688 leader_election.cc:302] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710e390f949e475da4fa7c8eb1a71d25, faaa573976c645998b868a2f02157543; no voters:
I20260928 13:10:58.329770 21838 raft_consensus.cc:2781] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:10:58.329916 21838 raft_consensus.cc:686] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [term 1 LEADER]: Becoming Leader. State: Replica: faaa573976c645998b868a2f02157543, State: Running, Role: LEADER
I20260928 13:10:58.330009 21838 consensus_queue.cc:237] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710e390f949e475da4fa7c8eb1a71d25" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44187 } } peers { permanent_uuid: "faaa573976c645998b868a2f02157543" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35693 } } peers { permanent_uuid: "e5dacf0a02174d248e2d1b9753c570f0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 32881 } }
I20260928 13:10:58.330852 21332 catalog_manager.cc:5495] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 reported cstate change: term changed from 0 to 1, leader changed from <none> to faaa573976c645998b868a2f02157543 (127.8.88.131). New cstate: current_term: 1 leader_uuid: "faaa573976c645998b868a2f02157543" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "710e390f949e475da4fa7c8eb1a71d25" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faaa573976c645998b868a2f02157543" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5dacf0a02174d248e2d1b9753c570f0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 32881 } health_report { overall_health: UNKNOWN } } }
I20260928 13:10:58.335246 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.321378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:59255 (local address 127.8.88.130:32881)
0928 13:10:58.321416 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:58.321418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:58.325655 (+ 4237us) server_negotiation.cc:407] Connection header received
0928 13:10:58.325761 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:58.325762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:58.325776 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:58.325806 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:10:58.326020 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.326751 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.334250 (+ 7499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.334825 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.334897 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:58.334908 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:58.335120 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 13:10:58.335166 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:10:58.335486 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.321204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:32881 (local address 127.8.88.131:59255)
0928 13:10:58.325598 (+ 4394us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:58.325610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:10:58.325684 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:10:58.325869 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:10:58.325870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:10:58.325872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:10:58.325937 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:58.325939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.331264 (+ 5325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.331265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:58.332660 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:10:58.332664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.334977 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.334978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:10:58.335044 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:58.335047 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:10:58.335375 (+ 328us) client_negotiation.cc:240] Negotiation successful
0928 13:10:58.335420 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4381,"thread_start_us":75,"threads_started":1}
I20260928 13:10:58.335820 21618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e0a0cad16f4d538d9adea0ee59a92e" candidate_uuid: "faaa573976c645998b868a2f02157543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5dacf0a02174d248e2d1b9753c570f0" is_pre_election: true
I20260928 13:10:58.336004 21618 raft_consensus.cc:2454] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faaa573976c645998b868a2f02157543 in term 0.
I20260928 13:10:58.336141 21617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e0a0cad16f4d538d9adea0ee59a92e" candidate_uuid: "faaa573976c645998b868a2f02157543" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5dacf0a02174d248e2d1b9753c570f0"
I20260928 13:10:58.336212 21617 raft_consensus.cc:3037] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:10:58.337222 21617 raft_consensus.cc:2454] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faaa573976c645998b868a2f02157543 in term 1.
I20260928 13:10:58.340947 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.336634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.8.88.131:35693)
0928 13:10:58.336689 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 13:10:58.336692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:10:58.337548 (+ 856us) server_negotiation.cc:407] Connection header received
0928 13:10:58.337623 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:10:58.337624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:10:58.337645 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:10:58.337718 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:10:58.337902 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.338592 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.339695 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:10:58.340006 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:10:58.340061 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:10:58.340384 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:10:58.340518 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:10:58.340586 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 13:10:58.340817 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 13:10:58.340839 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:10:58.343542 21617 raft_consensus.cc:1261] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0 [term 1 FOLLOWER]: Refusing update from remote peer faaa573976c645998b868a2f02157543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:10:58.343938 21842 consensus_queue.cc:1035] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5dacf0a02174d248e2d1b9753c570f0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 32881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:10:58.346566 21484 raft_consensus.cc:1261] T e1e0a0cad16f4d538d9adea0ee59a92e P 710e390f949e475da4fa7c8eb1a71d25 [term 1 FOLLOWER]: Refusing update from remote peer faaa573976c645998b868a2f02157543: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:10:58.346940 21842 consensus_queue.cc:1035] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [LEADER]: Connected to new peer: Peer: permanent_uuid: "710e390f949e475da4fa7c8eb1a71d25" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:10:58.351298 21845 mvcc.cc:205] Tried to move back new op lower bound from 7334301934971895808 to 7334301934920318976. Current Snapshot: MvccSnapshot[applied={T|T < 7334301934971895808}]
I20260928 13:10:58.351861 21848 mvcc.cc:205] Tried to move back new op lower bound from 7334301934971895808 to 7334301934920318976. Current Snapshot: MvccSnapshot[applied={T|T < 7334301934971895808}]
I20260928 13:10:58.353906 21846 mvcc.cc:205] Tried to move back new op lower bound from 7334301934971895808 to 7334301934920318976. Current Snapshot: MvccSnapshot[applied={T|T < 7334301934971895808}]
I20260928 13:10:58.356339 21332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:10:58.356432 21332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:10:58.357216 21332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01017f7c9e5f4ed48e150b049ce0c458: Sending DeleteTablet for 3 replicas of tablet e1e0a0cad16f4d538d9adea0ee59a92e
I20260928 13:10:58.357882 21464 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e0a0cad16f4d538d9adea0ee59a92e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:10:58 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:36750
I20260928 13:10:58.358139 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e0a0cad16f4d538d9adea0ee59a92e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:10:58 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:54220
I20260928 13:10:58.358366 21598 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e0a0cad16f4d538d9adea0ee59a92e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:10:58 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:41746
I20260928 13:10:58.359004 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 21370
I20260928 13:10:58.369055 21855 tablet_replica.cc:330] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543: stopping tablet replica
I20260928 13:10:58.369246 21855 raft_consensus.cc:2229] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:10:58.369417 21855 raft_consensus.cc:2258] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:10:58.369618 21856 tablet_replica.cc:330] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0: stopping tablet replica
I20260928 13:10:58.369755 21856 raft_consensus.cc:2229] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:10:58.369879 21856 raft_consensus.cc:2258] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:10:58.370443 21855 ts_tablet_manager.cc:1905] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:10:58.381049 21855 ts_tablet_manager.cc:1918] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:10:58.381172 21855 log.cc:1196] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/e1e0a0cad16f4d538d9adea0ee59a92e
I20260928 13:10:58.381309 21856 ts_tablet_manager.cc:1905] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:10:58.381546 21855 ts_tablet_manager.cc:1939] T e1e0a0cad16f4d538d9adea0ee59a92e P faaa573976c645998b868a2f02157543: Deleting consensus metadata
I20260928 13:10:58.382083 21319 catalog_manager.cc:4841] TS faaa573976c645998b868a2f02157543 (127.8.88.131:35693): tablet e1e0a0cad16f4d538d9adea0ee59a92e (table test-table [id=6c32f4af95c440f9864772f258f39587]) successfully deleted
W20260928 13:10:58.382344 21319 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:44187 (error 108)
I20260928 13:10:58.383129 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 21534
I20260928 13:10:58.383292 21856 ts_tablet_manager.cc:1918] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:10:58.383368 21856 log.cc:1196] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/e1e0a0cad16f4d538d9adea0ee59a92e
I20260928 13:10:58.383594 21856 ts_tablet_manager.cc:1939] T e1e0a0cad16f4d538d9adea0ee59a92e P e5dacf0a02174d248e2d1b9753c570f0: Deleting consensus metadata
I20260928 13:10:58.391472 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 21667
I20260928 13:10:58.393041 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.392848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:44187 (local address 127.0.0.1:36766)
0928 13:10:58.392931 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:58.392991 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:44187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 13:10:58.393167 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:10:58.393008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:32881 (local address 127.0.0.1:41752)
0928 13:10:58.393124 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 13:10:58.393144 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.130:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260928 13:10:58.393296 21319 catalog_manager.cc:4584] TS 710e390f949e475da4fa7c8eb1a71d25 (127.8.88.129:44187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1e0a0cad16f4d538d9adea0ee59a92e: Network error: Client connection negotiation failed: client connection to 127.8.88.129:44187: connect: Connection refused (error 111)
I20260928 13:10:58.398212 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 21299
I20260928 13:10:58.405689 8546 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 13:10:58.750574 8546 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 13:10:58.750690 8546 mini_ranger.cc:66] Stopping Ranger...
I20260928 13:10:59.096001 8546 mini_ranger.cc:68] Stopped Ranger
2026-09-28 13:10:59.096 UTC [20366] LOG: received smart shutdown request
2026-09-28 13:10:59.100 UTC [20366] LOG: background worker "logical replication launcher" (PID 20375) exited with exit code 1
2026-09-28 13:10:59.100 UTC [20370] LOG: shutting down
2026-09-28 13:10:59.320 UTC [20366] LOG: database system is shut down
2026-09-28T13:10:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50450 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:10:59 dist-test-slave-p1lc krb5kdc[21866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:10:59 dist-test-slave-p1lc krb5kdc[21866](info): set up 2 sockets
Sep 28 13:10:59 dist-test-slave-p1lc krb5kdc[21866](info): commencing operation
krb5kdc: starting...
W20260928 13:11:00.496856 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:11:00 dist-test-slave-p1lc krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:11:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:11:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:11:00.625351 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:38305
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:42743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:38305,127.8.88.189:34667,127.8.88.188:42563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:00.708320 21879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:00.708549 21879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:00.708575 21879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:00.709815 21879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:11:00.709862 21879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:00.709877 21879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:00.709890 21879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:11:00.709903 21879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:11:00.711499 21879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:42743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:38305,127.8.88.189:34667,127.8.88.188:42563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:38305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:00.711773 21879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:00.712025 21879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:00.717393 21884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:00.717394 21885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:00.717406 21887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:00.717758 21879 server_base.cc:1034] running on GCE node
I20260928 13:11:00.718044 21879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:00.718364 21879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:00.719539 21879 hybrid_clock.cc:648] HybridClock initialized: now 1790601060719508 us; error 54 us; skew 500 ppm
Sep 28 13:11:00 dist-test-slave-p1lc krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:00.727089 21879 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:11:00.727536 21879 webserver.cc:460] Webserver started at http://127.8.88.190:39871/ using document root <none> and password file <none>
I20260928 13:11:00.727807 21879 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:00.727857 21879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:00.727998 21879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:11:00.729197 21879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "90d64a8c83ec40e8a4c621e88077c7f1"
format_stamp: "Formatted at 2026-09-28 13:11:00 on dist-test-slave-p1lc"
server_key: "1691c7e3656542608fa1a531e75d389e"
server_key_iv: "e46fc94523c06c8f9c6ad6e7b9336bd9"
server_key_version: "encryptionkey@0"
I20260928 13:11:00.729586 21879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "90d64a8c83ec40e8a4c621e88077c7f1"
format_stamp: "Formatted at 2026-09-28 13:11:00 on dist-test-slave-p1lc"
server_key: "1691c7e3656542608fa1a531e75d389e"
server_key_iv: "e46fc94523c06c8f9c6ad6e7b9336bd9"
server_key_version: "encryptionkey@0"
I20260928 13:11:00.731040 21879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:11:00.731874 21894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:00.732075 21879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:00.732172 21879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "90d64a8c83ec40e8a4c621e88077c7f1"
format_stamp: "Formatted at 2026-09-28 13:11:00 on dist-test-slave-p1lc"
server_key: "1691c7e3656542608fa1a531e75d389e"
server_key_iv: "e46fc94523c06c8f9c6ad6e7b9336bd9"
server_key_version: "encryptionkey@0"
I20260928 13:11:00.732249 21879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:00.749742 21879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:00.752511 21879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:00.752648 21879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:00.757036 21879 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:38305
I20260928 13:11:00.757076 21946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:38305 every 8 connection(s)
I20260928 13:11:00.757493 21879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:11:00.758553 21947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:00.760190 21947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:11:00.760659 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 21879
I20260928 13:11:00.760780 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:11:00.761014 8546 external_mini_cluster.cc:1428] Setting key 3cbbedc94f4f684aa58b8f1bcd7712b4
Sep 28 13:11:00 dist-test-slave-p1lc krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.88.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.189@KRBTEST.COM" created.
I20260928 13:11:00.773582 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.761137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:38305 (local address 127.0.0.1:44746)
0928 13:11:00.761345 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.761395 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:00.762145 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:00.762507 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:00.762509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:00.762832 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:00.763000 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:00.763005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.764492 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.764494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:00.765583 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:00.765587 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.766421 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.766422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:00.766493 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:00.767144 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:00.767152 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:00.769247 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:00.771484 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:00.771491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:00.771493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:00.771800 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:00.772122 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:00.772123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:00.772124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:00.772256 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:00.772914 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:00.772988 (+ 74us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:00.773095 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 13:11:00.773201 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 13:11:00.773390 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260928 13:11:00.773988 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.761383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44746 (local address 127.8.88.190:38305)
0928 13:11:00.761533 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:00.761537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:00.761566 (+ 29us) server_negotiation.cc:407] Connection header received
0928 13:11:00.762359 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:00.762361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:00.762379 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:00.762501 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:00.763104 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.764376 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.765686 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.766293 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.766449 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:00.769347 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:00.769358 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:00.769358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:00.769368 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:00.771357 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:00.771897 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:00.771900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:00.771901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:00.771991 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:00.772367 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:00.772368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:00.772368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:00.772765 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:00.773704 (+ 939us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:00.773788 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 13:11:00.773873 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260928 13:11:00.775911 21947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:11:00.776402 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.776268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:34667 (local address 127.0.0.1:41748)
0928 13:11:00.776319 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.776368 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 13:11:00.776533 21896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.88.189:34667: connect: Connection refused (error 111)
I20260928 13:11:00.777088 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.776979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:34667 (local address 127.0.0.1:41750)
0928 13:11:00.777033 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.777068 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 13:11:00.777299 21947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.189:34667: Network error: Client connection negotiation failed: client connection to 127.8.88.189:34667: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.189.keytab.
Entry for principal kudu/127.8.88.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.189.keytab.
I20260928 13:11:00.793205 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.189:34667
--webserver_interface=127.8.88.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.189.keytab
--principal=kudu/127.8.88.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:42743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:38305,127.8.88.189:34667,127.8.88.188:42563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:11:00.826586 21947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } attempt: 1
I20260928 13:11:00.827222 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.827041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:34667 (local address 127.0.0.1:41762)
0928 13:11:00.827136 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.827181 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 13:11:00.827638 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.827511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:34667 (local address 127.0.0.1:41772)
0928 13:11:00.827561 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.827598 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 13:11:00.827744 21947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.189:34667: Network error: Client connection negotiation failed: client connection to 127.8.88.189:34667: connect: Connection refused (error 111)
W20260928 13:11:00.881824 21954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:00.882097 21954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:00.882136 21954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:00.883416 21954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:11:00.883496 21954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:00.883523 21954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:00.883558 21954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:11:00.883580 21954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:11:00.885347 21954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:42743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:38305,127.8.88.189:34667,127.8.88.188:42563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.189.keytab
--principal=kudu/127.8.88.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.189:34667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.88.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:00.885658 21954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:00.885905 21954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:00.891875 21959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:00.891851 21962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:00.891896 21960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:00.892228 21954 server_base.cc:1034] running on GCE node
I20260928 13:11:00.892645 21954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:00.893059 21954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:00.894285 21954 hybrid_clock.cc:648] HybridClock initialized: now 1790601060894293 us; error 55 us; skew 500 ppm
I20260928 13:11:00.895941 21947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } attempt: 2
I20260928 13:11:00.896441 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.896238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:34667 (local address 127.0.0.1:41784)
0928 13:11:00.896315 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.896408 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 13:11:00.896981 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.896772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:34667 (local address 127.0.0.1:41796)
0928 13:11:00.896858 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.896948 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:34667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
Sep 28 13:11:00 dist-test-slave-p1lc krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 13:11:00.897135 21947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.189:34667: Network error: Client connection negotiation failed: client connection to 127.8.88.189:34667: connect: Connection refused (error 111)
I20260928 13:11:00.897503 21954 init.cc:377] Logged in from keytab as kudu/127.8.88.189@KRBTEST.COM (short username kudu)
I20260928 13:11:00.898072 21954 webserver.cc:460] Webserver started at http://127.8.88.189:41383/ using document root <none> and password file <none>
I20260928 13:11:00.898350 21954 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:00.898401 21954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:00.898556 21954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:11:00.899796 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/data/instance:
uuid: "c028a6ef889e4d04bbe9fe116f404f61"
format_stamp: "Formatted at 2026-09-28 13:11:00 on dist-test-slave-p1lc"
server_key: "6abd2819bc5c4dcf29244147d22f2ee8"
server_key_iv: "70f16acb67d58373f571fdc322d5df17"
server_key_version: "encryptionkey@0"
I20260928 13:11:00.900206 21954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/wal/instance:
uuid: "c028a6ef889e4d04bbe9fe116f404f61"
format_stamp: "Formatted at 2026-09-28 13:11:00 on dist-test-slave-p1lc"
server_key: "6abd2819bc5c4dcf29244147d22f2ee8"
server_key_iv: "70f16acb67d58373f571fdc322d5df17"
server_key_version: "encryptionkey@0"
I20260928 13:11:00.901862 21954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:11:00.902724 21969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:00.902954 21954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:11:00.903067 21954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/wal
uuid: "c028a6ef889e4d04bbe9fe116f404f61"
format_stamp: "Formatted at 2026-09-28 13:11:00 on dist-test-slave-p1lc"
server_key: "6abd2819bc5c4dcf29244147d22f2ee8"
server_key_iv: "70f16acb67d58373f571fdc322d5df17"
server_key_version: "encryptionkey@0"
I20260928 13:11:00.903147 21954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:00.923983 21954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:00.926270 21954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:00.926451 21954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:00.930932 21954 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.189:34667
I20260928 13:11:00.930974 22021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.189:34667 every 8 connection(s)
I20260928 13:11:00.931354 21954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/data/info.pb
I20260928 13:11:00.932260 22022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:00.933996 22022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:11:00.939662 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 21954
I20260928 13:11:00.939808 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-1/wal/instance
I20260928 13:11:00.940004 8546 external_mini_cluster.cc:1428] Setting key 40970233967667e5030e6b6df80504c2
Sep 28 13:11:00 dist-test-slave-p1lc krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.189@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:00.944468 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.934711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44750 (local address 127.8.88.190:38305)
0928 13:11:00.934777 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:00.934779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:00.935016 (+ 237us) server_negotiation.cc:407] Connection header received
0928 13:11:00.935603 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:00.935605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:00.935617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:00.935669 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:00.936347 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.936853 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.937933 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.938147 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.938241 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:00.940718 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:00.940720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:00.940720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:00.940729 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:00.942648 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:00.943315 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:00.943316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:00.943317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:00.943392 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:00.943737 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:00.943738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:00.943739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:00.943870 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:00.943933 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:00.944324 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 13:11:00.944400 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 13:11:00.944761 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.934676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:38305 (local address 127.0.0.1:44750)
0928 13:11:00.934893 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.934930 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:00.935472 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:00.935749 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:00.935753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:00.936026 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:00.936235 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:00.936239 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.937005 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.937006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:00.937839 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:00.937841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.938254 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.938255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:00.938391 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:00.938911 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:00.938919 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:00.940613 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:00.942756 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:00.942765 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:00.942767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:00.943141 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:00.943482 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:00.943483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:00.943483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:00.943641 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:00.943949 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:00.943957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:00.944154 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 13:11:00.944297 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 13:11:00.944494 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260928 13:11:00.945406 22022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.88.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.188@KRBTEST.COM" created.
Sep 28 13:11:00 dist-test-slave-p1lc krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.189@KRBTEST.COM for kudu/127.8.88.189@KRBTEST.COM
I20260928 13:11:00.955708 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.945719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:34667 (local address 127.0.0.1:41808)
0928 13:11:00.945771 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.945781 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:00.945818 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:00.946108 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:00.946109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:00.946264 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:00.946324 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:00.946326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.947204 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.947205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:00.947709 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:00.947711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.948229 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.948230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:00.948298 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:00.948702 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:00.948706 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:00.952107 (+ 3401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:00.954544 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:00.954545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:00.954546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:00.954711 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:00.954977 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:00.954978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:00.954978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:00.955028 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:00.955364 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:00.955364 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:00.955401 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 13:11:00.955452 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 13:11:00.955532 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 13:11:00.955826 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.945802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.8.88.189:34667)
0928 13:11:00.945976 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:00.945980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:00.945996 (+ 16us) server_negotiation.cc:407] Connection header received
0928 13:11:00.946013 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:00.946014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:00.946025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:00.946108 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:00.946405 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.947102 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.947796 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.948146 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.948229 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:00.952205 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:00.952213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:00.952214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:00.952220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:00.954452 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:00.954820 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:00.954824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:00.954824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:00.954901 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:00.955088 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:00.955089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:00.955089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:00.955294 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:00.955387 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:00.955491 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 13:11:00.955610 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
I20260928 13:11:00.956507 22022 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:11:00.956977 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.956793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:42563 (local address 127.0.0.1:56984)
0928 13:11:00.956851 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.956938 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 13:11:00.957149 21970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
I20260928 13:11:00.957579 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.957493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:42563 (local address 127.0.0.1:56994)
0928 13:11:00.957539 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.957564 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 13:11:00.957729 22022 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:42563: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.188.keytab.
Entry for principal kudu/127.8.88.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.188.keytab.
I20260928 13:11:00.969421 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.188:42563
--webserver_interface=127.8.88.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.188.keytab
--principal=kudu/127.8.88.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:42743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:38305,127.8.88.189:34667,127.8.88.188:42563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:11:00.988319 21947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } attempt: 3
Sep 28 13:11:00 dist-test-slave-p1lc krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for kudu/127.8.88.189@KRBTEST.COM
I20260928 13:11:00.995883 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.988828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.8.88.189:34667)
0928 13:11:00.988928 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:00.988930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:00.988944 (+ 14us) server_negotiation.cc:407] Connection header received
0928 13:11:00.989027 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:00.989028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:00.989041 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:00.989112 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:00.989446 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.989951 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.990685 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.990906 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.990982 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:00.993222 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:00.993224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:00.993224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:00.993231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:00.994737 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:00.995145 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:00.995146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:00.995147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:00.995203 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:00.995411 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:00.995412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:00.995412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:00.995528 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:00.995580 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:00.995736 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 13:11:00.995812 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 13:11:00.995923 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.988760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:34667 (local address 127.0.0.1:41818)
0928 13:11:00.988827 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.988837 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:00.988966 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:00.989130 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:00.989131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:00.989299 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:00.989361 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:00.989363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.990043 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.990045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:00.990594 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:00.990596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:00.990991 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:00.990993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:00.991075 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:00.991558 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:00.991571 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:00.993092 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:00.994830 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:00.994831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:00.994832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:00.995064 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:00.995272 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:00.995273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:00.995274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:00.995341 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:00.995590 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:00.995591 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:00.995647 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 13:11:00.995721 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 13:11:00.995825 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 13:11:00.996618 21947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:11:00.997008 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.996897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:42563 (local address 127.0.0.1:57002)
0928 13:11:00.996937 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.996980 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 13:11:00.997376 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:00.997283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:42563 (local address 127.0.0.1:57018)
0928 13:11:00.997328 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:00.997359 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 13:11:00.997485 21947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:42563: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
I20260928 13:11:01.006965 22022 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } attempt: 1
I20260928 13:11:01.007457 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.007311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:42563 (local address 127.0.0.1:57022)
0928 13:11:01.007386 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:01.007428 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 13:11:01.007879 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.007724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:42563 (local address 127.0.0.1:57036)
0928 13:11:01.007788 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:01.007838 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260928 13:11:01.008024 22022 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:42563: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
I20260928 13:11:01.028652 21947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } attempt: 1
I20260928 13:11:01.029186 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.029047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:42563 (local address 127.0.0.1:57048)
0928 13:11:01.029115 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:01.029157 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 13:11:01.029618 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.029520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:42563 (local address 127.0.0.1:57050)
0928 13:11:01.029566 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:01.029594 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 13:11:01.029706 21947 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:42563: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
W20260928 13:11:01.056576 22029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:01.056829 22029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:01.056854 22029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:01.058207 22029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:11:01.058260 22029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:01.058282 22029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:01.058296 22029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:11:01.058308 22029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:11:01.059973 22029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:42743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:38305,127.8.88.189:34667,127.8.88.188:42563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.188.keytab
--principal=kudu/127.8.88.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.188:42563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.88.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:01.060268 22029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:01.060540 22029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:01.065696 22037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:01.065659 22035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:01.065641 22034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:01.065923 22029 server_base.cc:1034] running on GCE node
I20260928 13:11:01.066269 22029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:01.066628 22029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:01.067839 22029 hybrid_clock.cc:648] HybridClock initialized: now 1790601061067801 us; error 41 us; skew 500 ppm
Sep 28 13:11:01 dist-test-slave-p1lc krb5kdc[21866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:01.070674 22029 init.cc:377] Logged in from keytab as kudu/127.8.88.188@KRBTEST.COM (short username kudu)
I20260928 13:11:01.071223 22029 webserver.cc:460] Webserver started at http://127.8.88.188:33197/ using document root <none> and password file <none>
I20260928 13:11:01.071491 22029 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:01.071542 22029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:01.071674 22029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:11:01.072850 22029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/data/instance:
uuid: "2929db0e30e54e1087ee6079303c6f15"
format_stamp: "Formatted at 2026-09-28 13:11:01 on dist-test-slave-p1lc"
server_key: "a687c347d5386afc8cc4658a14ee3343"
server_key_iv: "bcf08211e06efb019dd224be31626b60"
server_key_version: "encryptionkey@0"
I20260928 13:11:01.073311 22029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/wal/instance:
uuid: "2929db0e30e54e1087ee6079303c6f15"
format_stamp: "Formatted at 2026-09-28 13:11:01 on dist-test-slave-p1lc"
server_key: "a687c347d5386afc8cc4658a14ee3343"
server_key_iv: "bcf08211e06efb019dd224be31626b60"
server_key_version: "encryptionkey@0"
I20260928 13:11:01.074790 22029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:01.075865 22044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:01.076153 22029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:01.076274 22022 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } attempt: 2
I20260928 13:11:01.076370 22029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/wal
uuid: "2929db0e30e54e1087ee6079303c6f15"
format_stamp: "Formatted at 2026-09-28 13:11:01 on dist-test-slave-p1lc"
server_key: "a687c347d5386afc8cc4658a14ee3343"
server_key_iv: "bcf08211e06efb019dd224be31626b60"
server_key_version: "encryptionkey@0"
I20260928 13:11:01.076527 22029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:01.076805 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.076603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:42563 (local address 127.0.0.1:57064)
0928 13:11:01.076679 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:01.076766 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 13:11:01.077209 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.077113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:42563 (local address 127.0.0.1:57066)
0928 13:11:01.077158 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:01.077186 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 13:11:01.077343 22022 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:42563: Network error: Client connection negotiation failed: client connection to 127.8.88.188:42563: connect: Connection refused (error 111)
I20260928 13:11:01.089512 22029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:01.092109 22029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:01.092252 22029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:01.097038 22029 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.188:42563
I20260928 13:11:01.097153 22096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.188:42563 every 8 connection(s)
I20260928 13:11:01.097496 22029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/data/info.pb
I20260928 13:11:01.098461 22097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:01.100231 22097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:11:01.104743 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 22029
I20260928 13:11:01.104926 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790600874459249-8546-0/minicluster-data/master-2/wal/instance
I20260928 13:11:01.104911 21947 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } attempt: 2
I20260928 13:11:01.105155 8546 external_mini_cluster.cc:1428] Setting key 8cade96dff1240d6a6ee4fa03ec41969
Sep 28 13:11:01 dist-test-slave-p1lc krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.188@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
Sep 28 13:11:01 dist-test-slave-p1lc krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for kudu/127.8.88.188@KRBTEST.COM
Sep 28 13:11:01 dist-test-slave-p1lc krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.188@KRBTEST.COM
I20260928 13:11:01.112809 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.101094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.8.88.190:38305)
0928 13:11:01.101186 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:01.101188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:01.101423 (+ 235us) server_negotiation.cc:407] Connection header received
0928 13:11:01.101923 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:01.101924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:01.101935 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:01.101996 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:01.102641 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.103159 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.104212 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.104447 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.104524 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.107823 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:01.107825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:01.107825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:01.107833 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:01.110676 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.111318 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.111319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.111319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.111386 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.111927 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.111928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.111929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.112101 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:01.112240 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:01.112586 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 13:11:01.112682 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 13:11:01.113351 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.101068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:38305 (local address 127.0.0.1:44762)
0928 13:11:01.101287 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:01.101335 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:01.101809 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:01.102059 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:01.102062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:01.102333 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:01.102533 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:01.102537 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.103262 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.103263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:01.104094 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:01.104097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.104576 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.104578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:01.104772 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.105723 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:01.105744 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:01.107715 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:01.110787 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:01.110810 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:01.110813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:01.111246 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:01.111475 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:01.111476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:01.111477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:01.111628 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:01.112262 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:01.112276 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:01.112420 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 13:11:01.112537 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 13:11:01.112784 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":113,"threads_started":1}
I20260928 13:11:01.114063 22097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:11:01.114252 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.105215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:42563 (local address 127.0.0.1:57074)
0928 13:11:01.105362 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:01.105374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:01.105422 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:01.105763 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:01.105764 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:01.105986 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:01.106031 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:01.106033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.107120 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.107121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:01.107894 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:01.107897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.108508 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.108509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:01.108572 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.109018 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:01.109023 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:01.110657 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:01.112996 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:01.112997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:01.112998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:01.113242 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:01.113519 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:01.113520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:01.113520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:01.113582 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:01.113919 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:01.113920 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:01.113970 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 13:11:01.114042 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 13:11:01.114150 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 13:11:01.114787 22100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.105304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57074 (local address 127.8.88.188:42563)
0928 13:11:01.105525 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:01.105530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:01.105553 (+ 23us) server_negotiation.cc:407] Connection header received
0928 13:11:01.105579 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:01.105581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:01.105627 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:01.106097 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:01.106119 (+ 22us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.107026 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.107981 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.108437 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.108511 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.110825 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:01.110833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:01.110834 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:01.110840 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:01.112859 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.113355 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.113359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.113359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.113434 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.113658 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.113658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.113659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.113843 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:01.113925 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:01.114109 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 13:11:01.114467 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":114,"threads_started":1}
I20260928 13:11:01.115950 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.105586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.8.88.188:42563)
0928 13:11:01.105720 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:01.105722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:01.105803 (+ 81us) server_negotiation.cc:407] Connection header received
0928 13:11:01.105889 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:01.105890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:01.105900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:01.105972 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:01.106755 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.107442 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.108373 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.108684 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.108764 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.112671 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:01.112672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:01.112673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:01.112693 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:01.114599 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.115016 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.115017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.115017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.115073 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.115273 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.115274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.115274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.115402 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:01.115531 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:01.115744 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 13:11:01.115828 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260928 13:11:01.116593 21947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1: Bootstrap starting.
I20260928 13:11:01.117476 21947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:01.117847 21947 log.cc:824] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:01.118691 21947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1: No bootstrap required, opened a new log
Sep 28 13:11:01 dist-test-slave-p1lc krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.188@KRBTEST.COM for kudu/127.8.88.189@KRBTEST.COM
I20260928 13:11:01.121187 21947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } }
I20260928 13:11:01.121352 21947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:01.121418 21947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d64a8c83ec40e8a4c621e88077c7f1, State: Initialized, Role: FOLLOWER
I20260928 13:11:01.121587 21947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } }
I20260928 13:11:01.121922 22104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } } }
I20260928 13:11:01.121984 21947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 13:11:01 dist-test-slave-p1lc krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
W20260928 13:11:01.124085 22115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:11:01.124140 22115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:11:01.124202 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.114435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.8.88.189:34667)
0928 13:11:01.114492 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:01.114495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:01.114943 (+ 448us) server_negotiation.cc:407] Connection header received
0928 13:11:01.115043 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:01.115044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:01.115057 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:01.115086 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:01.115498 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.116164 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.116975 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.117236 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.117273 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.119391 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:01.119393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:01.119393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:01.119401 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:01.122179 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.122701 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.122702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.122703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.122774 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.123326 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.123328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.123328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.123489 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:01.123541 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:01.124023 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 13:11:01.124107 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 13:11:01.124516 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.114324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:34667 (local address 127.0.0.1:41820)
0928 13:11:01.114890 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:01.114900 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:01.114969 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:01.115149 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:01.115150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:01.115314 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:01.115381 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:01.115383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.116251 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.116252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:01.116826 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:01.116828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.117375 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.117376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:01.117472 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.117890 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:01.117896 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:01.119296 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:01.122394 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:01.122395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:01.122395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:01.122620 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:01.122868 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:01.122869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:01.122870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:01.122950 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:01.123597 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:01.123598 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:01.123890 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 13:11:01.124288 (+ 398us) client_negotiation.cc:240] Negotiation successful
0928 13:11:01.124401 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550}
I20260928 13:11:01.122078 22104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:11:01.126703 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.117647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.8.88.190:38305)
0928 13:11:01.117688 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:01.117690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:01.117698 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:11:01.117762 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:01.117763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:01.117773 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:01.117821 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:01.118424 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.119065 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.119853 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.120063 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.120138 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.122799 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:01.122801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:01.122801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:01.122808 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:01.124504 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.125585 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.125586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.125586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.125646 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.125856 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.125857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.125857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.125981 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:01.126128 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:01.126532 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 13:11:01.126616 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:11:01.124936 21879 master_runner.cc:186] Error getting master registration for 127.8.88.189:34667: OK, instance_id { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" instance_seqno: 1790601060929597 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 13:11:01.127206 21879 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 13:11:01.127267 22097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:11:01 dist-test-slave-p1lc krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.188@KRBTEST.COM for kudu/127.8.88.188@KRBTEST.COM
Sep 28 13:11:01 dist-test-slave-p1lc krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.189@KRBTEST.COM
I20260928 13:11:01.137594 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.127617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.8.88.188:42563)
0928 13:11:01.127675 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:01.127677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:01.127685 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:11:01.127706 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:01.127707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:01.127720 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:01.127781 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:01.128140 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.129080 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.130191 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.130519 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.130589 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.133409 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:01.133410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:01.133410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:01.133416 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:01.136160 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.136645 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.136646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.136659 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.136715 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.136954 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.136955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.136955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.137157 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:01.137197 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:01.137391 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 13:11:01.137484 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 13:11:01.137992 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.127526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:42563 (local address 127.0.0.1:57092)
0928 13:11:01.127570 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:01.127580 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:01.127615 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:01.127819 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:01.127820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:01.127981 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:01.128022 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:01.128024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.129224 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.129323 (+ 99us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:01.130094 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:01.130097 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.130614 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.130615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:01.130701 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.131710 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:01.131716 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:01.133313 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:01.136274 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:01.136276 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:01.136276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:01.136534 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:01.136802 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:01.136802 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:01.136803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:01.136858 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:01.137239 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:01.137240 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:01.137279 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 13:11:01.137727 (+ 448us) client_negotiation.cc:240] Negotiation successful
0928 13:11:01.137858 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 13:11:01.139041 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.128048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.8.88.189:34667)
0928 13:11:01.128119 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:01.128121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:01.128128 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:11:01.128171 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:01.128172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:01.128183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:01.128273 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:01.129356 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.130129 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.131092 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.131545 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.131622 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.134563 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:01.134564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:01.134564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:01.134572 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:01.137001 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.137407 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.137408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.137409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.137463 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.137875 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.137876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.137876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.138018 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:01.138252 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:01.138869 (+ 617us) server_negotiation.cc:299] Negotiation successful
0928 13:11:01.138953 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 13:11:01.139974 22097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15: Bootstrap starting.
I20260928 13:11:01.140465 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:11:01.140802 22097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:01.141179 22097 log.cc:824] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:01.142076 22097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15: No bootstrap required, opened a new log
I20260928 13:11:01.144408 22097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } }
I20260928 13:11:01.144574 22097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:01.144616 22097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2929db0e30e54e1087ee6079303c6f15, State: Initialized, Role: FOLLOWER
I20260928 13:11:01.144767 22097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } }
I20260928 13:11:01.145108 22097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 13:11:01.149012 22136 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:11:01.149072 22136 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:11:01.150080 22125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } } }
I20260928 13:11:01.150243 22125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:11:01.149828 22029 master_runner.cc:186] Error getting master registration for 127.8.88.189:34667: OK, instance_id { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" instance_seqno: 1790601060929597 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 13:11:01.152124 22029 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 13:11:01.153119 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.142023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44774 (local address 127.8.88.190:38305)
0928 13:11:01.142063 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:01.142065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:01.142614 (+ 549us) server_negotiation.cc:407] Connection header received
0928 13:11:01.142721 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:01.142722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:01.142733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:01.142765 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:01.143519 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.144236 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.145241 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.145563 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.145613 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.149561 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:01.149562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:01.149562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:01.149570 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:01.151564 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.151993 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.151994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.151994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.152051 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.152254 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.152255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.152256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.152392 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:01.152519 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:01.152928 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 13:11:01.153019 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:11:01.154441 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.142146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.8.88.188:42563)
0928 13:11:01.142185 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:01.142187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:01.142371 (+ 184us) server_negotiation.cc:407] Connection header received
0928 13:11:01.142469 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:01.142470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:01.142482 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:01.142544 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:01.143325 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.144078 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.145394 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.145689 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.145729 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.147441 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:01.147443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:01.147443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:01.147450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:01.149468 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.152826 (+ 3358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.152827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.152827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.152911 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.153809 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.153810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.153810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.153953 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:01.154024 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:01.154260 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 13:11:01.154349 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:11:01.155066 22026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.142346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41838 (local address 127.8.88.189:34667)
0928 13:11:01.142381 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:01.142383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:01.146073 (+ 3690us) server_negotiation.cc:407] Connection header received
0928 13:11:01.146129 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:01.146130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:01.146142 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:01.146195 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:01.147809 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.148519 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.149510 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.150412 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.150490 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.152200 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:01.152202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:01.152203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:01.152210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:01.153733 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.154046 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.154047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.154048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.154105 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.154348 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.154349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.154349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.154466 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:01.154671 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:01.154862 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 13:11:01.154944 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 13:11:01.168515 22022 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } attempt: 3
Sep 28 13:11:01 dist-test-slave-p1lc krb5kdc[21866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601060, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.189@KRBTEST.COM for kudu/127.8.88.188@KRBTEST.COM
I20260928 13:11:01.177830 22024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.168890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:42563 (local address 127.0.0.1:57114)
0928 13:11:01.168945 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:01.168959 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:01.169000 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:01.169170 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:01.169171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:01.169376 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:01.169432 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:01.169434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.170400 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.170401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:01.171269 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:01.171272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.172314 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.172316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:01.172383 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.172943 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:01.172949 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:01.174585 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:01.176589 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:01.176590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:01.176590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:01.176826 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:01.177101 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:01.177102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:01.177103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:01.177168 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:01.177493 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:01.177494 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:01.177549 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 13:11:01.177616 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 13:11:01.177726 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 13:11:01.177855 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:01.168985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57114 (local address 127.8.88.188:42563)
0928 13:11:01.169030 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:01.169032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:01.169042 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:11:01.169069 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:01.169070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:01.169081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:01.169149 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:01.169518 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.170286 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.171379 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:01.171699 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:01.171794 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:01.174692 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:01.174694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:01.174694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:01.174703 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:01.176487 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.176950 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.176951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.176952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.177020 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:01.177259 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:01.177260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:01.177261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:01.177418 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:01.177488 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:01.177676 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 13:11:01.177759 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:11:01.181135 22022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61: Bootstrap starting.
I20260928 13:11:01.181895 22022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:01.182205 22022 log.cc:824] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:01.183073 22022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61: No bootstrap required, opened a new log
I20260928 13:11:01.185262 22022 raft_consensus.cc:348] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } }
I20260928 13:11:01.185441 22022 raft_consensus.cc:374] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:01.185480 22022 raft_consensus.cc:729] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c028a6ef889e4d04bbe9fe116f404f61, State: Initialized, Role: FOLLOWER
I20260928 13:11:01.185621 22022 consensus_queue.cc:260] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } }
I20260928 13:11:01.185984 22022 sys_catalog.cc:564] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:11:01.185945 22139 sys_catalog.cc:455] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } } }
I20260928 13:11:01.186136 22139 sys_catalog.cc:458] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 13:11:01.187945 22150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:11:01.188061 22150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:11:01.189888 21954 master_runner.cc:263] No leader master found from master c028a6ef889e4d04bbe9fe116f404f61
I20260928 13:11:01.189949 21954 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 13:11:01.253539 21900 tablet.cc:2369] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:11:01.289435 22125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:11:01.289628 22125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } }
I20260928 13:11:01.289932 22125 leader_election.cc:288] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d64a8c83ec40e8a4c621e88077c7f1 (127.8.88.190:38305), c028a6ef889e4d04bbe9fe116f404f61 (127.8.88.189:34667)
I20260928 13:11:01.290210 21921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2929db0e30e54e1087ee6079303c6f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" is_pre_election: true
I20260928 13:11:01.290409 21921 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2929db0e30e54e1087ee6079303c6f15 in term 0.
I20260928 13:11:01.290365 21996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2929db0e30e54e1087ee6079303c6f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c028a6ef889e4d04bbe9fe116f404f61" is_pre_election: true
I20260928 13:11:01.290488 21996 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2929db0e30e54e1087ee6079303c6f15 in term 0.
I20260928 13:11:01.290666 22046 leader_election.cc:302] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2929db0e30e54e1087ee6079303c6f15, c028a6ef889e4d04bbe9fe116f404f61; no voters:
I20260928 13:11:01.290886 22125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:11:01.290936 22125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:11:01.290959 22125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:01.291883 22125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } }
I20260928 13:11:01.292008 22125 leader_election.cc:288] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [CANDIDATE]: Term 1 election: Requested vote from peers 90d64a8c83ec40e8a4c621e88077c7f1 (127.8.88.190:38305), c028a6ef889e4d04bbe9fe116f404f61 (127.8.88.189:34667)
I20260928 13:11:01.292165 21996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2929db0e30e54e1087ee6079303c6f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c028a6ef889e4d04bbe9fe116f404f61"
I20260928 13:11:01.292238 21996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:01.292222 21921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2929db0e30e54e1087ee6079303c6f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d64a8c83ec40e8a4c621e88077c7f1"
I20260928 13:11:01.292372 21921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:01.293135 21996 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2929db0e30e54e1087ee6079303c6f15 in term 1.
I20260928 13:11:01.293290 21921 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2929db0e30e54e1087ee6079303c6f15 in term 1.
I20260928 13:11:01.293305 22046 leader_election.cc:302] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2929db0e30e54e1087ee6079303c6f15, c028a6ef889e4d04bbe9fe116f404f61; no voters:
I20260928 13:11:01.293377 22125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:11:01.293527 22125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [term 1 LEADER]: Becoming Leader. State: Replica: 2929db0e30e54e1087ee6079303c6f15, State: Running, Role: LEADER
I20260928 13:11:01.293618 22125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } }
I20260928 13:11:01.294085 22152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2929db0e30e54e1087ee6079303c6f15. Latest consensus state: current_term: 1 leader_uuid: "2929db0e30e54e1087ee6079303c6f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } } }
I20260928 13:11:01.294173 22152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:01.294371 22154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:11:01.294575 22154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:11:01.295858 21996 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [term 1 FOLLOWER]: Refusing update from remote peer 2929db0e30e54e1087ee6079303c6f15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:11:01.295872 21921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [term 1 FOLLOWER]: Refusing update from remote peer 2929db0e30e54e1087ee6079303c6f15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:11:01.296173 22152 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:11:01.296284 22125 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:11:01.297520 22104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2929db0e30e54e1087ee6079303c6f15. Latest consensus state: current_term: 1 leader_uuid: "2929db0e30e54e1087ee6079303c6f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } } }
I20260928 13:11:01.297528 22139 sys_catalog.cc:455] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2929db0e30e54e1087ee6079303c6f15. Latest consensus state: current_term: 1 leader_uuid: "2929db0e30e54e1087ee6079303c6f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } } }
I20260928 13:11:01.297631 22104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:11:01.297636 22139 sys_catalog.cc:458] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:11:01.299839 22152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2929db0e30e54e1087ee6079303c6f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } } }
I20260928 13:11:01.299950 22152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:01.300313 22125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2929db0e30e54e1087ee6079303c6f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } } }
I20260928 13:11:01.300386 22125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15 [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:01.300642 22155 mvcc.cc:205] Tried to move back new op lower bound from 7334301947065876480 to 7334301947059318784. Current Snapshot: MvccSnapshot[applied={T|T < 7334301947065876480}]
I20260928 13:11:01.300743 22157 mvcc.cc:205] Tried to move back new op lower bound from 7334301947065876480 to 7334301947059318784. Current Snapshot: MvccSnapshot[applied={T|T < 7334301947065876480}]
I20260928 13:11:01.300783 22139 sys_catalog.cc:455] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2929db0e30e54e1087ee6079303c6f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } } }
I20260928 13:11:01.300894 22139 sys_catalog.cc:458] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:11:01.301209 22104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2929db0e30e54e1087ee6079303c6f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d64a8c83ec40e8a4c621e88077c7f1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38305 } } peers { permanent_uuid: "c028a6ef889e4d04bbe9fe116f404f61" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 34667 } } peers { permanent_uuid: "2929db0e30e54e1087ee6079303c6f15" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 42563 } } }
I20260928 13:11:01.301306 22104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:11:01.301411 22156 mvcc.cc:205] Tried to move back new op lower bound from 7334301947065876480 to 7334301947059318784. Current Snapshot: MvccSnapshot[applied={T|T < 7334301947065876480}]
I20260928 13:11:01.301921 22154 catalog_manager.cc:1347] Generated new cluster ID: a4df14dd011b424ea1b8121abfe1565b
I20260928 13:11:01.301970 22154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:11:01.320294 22154 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:11:01.320950 22154 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:11:01.325136 22154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2929db0e30e54e1087ee6079303c6f15: Generated new TSK 0
I20260928 13:11:01.325352 22154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:11:02.124593 22115 catalog_manager.cc:1259] Loaded cluster ID: a4df14dd011b424ea1b8121abfe1565b
I20260928 13:11:02.124792 22115 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1: loading cluster ID for follower catalog manager: success
I20260928 13:11:02.125733 22115 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1: acquiring CA information for follower catalog manager: success
I20260928 13:11:02.126060 22115 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90d64a8c83ec40e8a4c621e88077c7f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:11:02.188550 22150 catalog_manager.cc:1259] Loaded cluster ID: a4df14dd011b424ea1b8121abfe1565b
I20260928 13:11:02.188632 22150 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61: loading cluster ID for follower catalog manager: success
I20260928 13:11:02.189514 22150 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61: acquiring CA information for follower catalog manager: success
I20260928 13:11:02.189836 22150 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c028a6ef889e4d04bbe9fe116f404f61: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:11:02.374750 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 21879
I20260928 13:11:02.380999 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 21954
I20260928 13:11:02.386760 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 22029
2026-09-28T13:11:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2943 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:11:02 dist-test-slave-p1lc krb5kdc[22168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:11:02 dist-test-slave-p1lc krb5kdc[22168](info): set up 2 sockets
Sep 28 13:11:02 dist-test-slave-p1lc krb5kdc[22168](info): commencing operation
krb5kdc: starting...
W20260928 13:11:03.433229 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:11:03 dist-test-slave-p1lc krb5kdc[22168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:11:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:11:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:11:03.561066 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:43721
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:43721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:03.646278 22181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:03.646539 22181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:03.646581 22181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:03.647756 22181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:11:03.647827 22181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:03.647854 22181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:03.647876 22181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:11:03.647907 22181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:11:03.649452 22181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:43721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:43721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:03.649734 22181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:03.649968 22181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:03.655329 22186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:03.655369 22187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:03.655419 22181 server_base.cc:1034] running on GCE node
W20260928 13:11:03.655360 22189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:03.655782 22181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:03.656152 22181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:03.657339 22181 hybrid_clock.cc:648] HybridClock initialized: now 1790601063657322 us; error 48 us; skew 500 ppm
Sep 28 13:11:03 dist-test-slave-p1lc krb5kdc[22168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601063, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:03.660115 22181 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:11:03.660554 22181 webserver.cc:460] Webserver started at http://127.8.88.190:42909/ using document root <none> and password file <none>
I20260928 13:11:03.660786 22181 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:03.660974 22181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:03.661134 22181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:11:03.662225 22181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "ab26adcde329419380eb61b4e02881bc"
format_stamp: "Formatted at 2026-09-28 13:11:03 on dist-test-slave-p1lc"
server_key: "0a13d46c3d0c8eb303bcada11ffe70f0"
server_key_iv: "008e714fafbb006ce2e4b0c72ef018a9"
server_key_version: "encryptionkey@0"
I20260928 13:11:03.662626 22181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "ab26adcde329419380eb61b4e02881bc"
format_stamp: "Formatted at 2026-09-28 13:11:03 on dist-test-slave-p1lc"
server_key: "0a13d46c3d0c8eb303bcada11ffe70f0"
server_key_iv: "008e714fafbb006ce2e4b0c72ef018a9"
server_key_version: "encryptionkey@0"
I20260928 13:11:03.664101 22181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:11:03.665000 22196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:03.665187 22181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:03.665294 22181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "ab26adcde329419380eb61b4e02881bc"
format_stamp: "Formatted at 2026-09-28 13:11:03 on dist-test-slave-p1lc"
server_key: "0a13d46c3d0c8eb303bcada11ffe70f0"
server_key_iv: "008e714fafbb006ce2e4b0c72ef018a9"
server_key_version: "encryptionkey@0"
I20260928 13:11:03.665395 22181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:03.675992 22181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:03.678388 22181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:03.678570 22181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:03.682593 22181 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:43721
I20260928 13:11:03.682631 22248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:43721 every 8 connection(s)
I20260928 13:11:03.683013 22181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:11:03.683962 22249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:03.686390 22249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc: Bootstrap starting.
I20260928 13:11:03.686589 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 22181
I20260928 13:11:03.686679 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:11:03.686893 8546 external_mini_cluster.cc:1428] Setting key 2039fe461726a4992996878b35d45ada
I20260928 13:11:03.687155 22249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:03.687506 22249 log.cc:824] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:03.688329 22249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc: No bootstrap required, opened a new log
I20260928 13:11:03.690735 22249 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab26adcde329419380eb61b4e02881bc" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43721 } }
I20260928 13:11:03.690948 22249 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:03.690999 22249 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab26adcde329419380eb61b4e02881bc, State: Initialized, Role: FOLLOWER
I20260928 13:11:03.691135 22249 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab26adcde329419380eb61b4e02881bc" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43721 } }
I20260928 13:11:03.691231 22249 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:11:03.691275 22249 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:11:03.691329 22249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:03.691993 22249 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab26adcde329419380eb61b4e02881bc" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43721 } }
I20260928 13:11:03.692098 22249 leader_election.cc:302] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab26adcde329419380eb61b4e02881bc; no voters:
I20260928 13:11:03.692261 22249 leader_election.cc:288] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:11:03.692482 22255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:11:03.692510 22249 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:11:03.692690 22255 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [term 1 LEADER]: Becoming Leader. State: Replica: ab26adcde329419380eb61b4e02881bc, State: Running, Role: LEADER
I20260928 13:11:03.692811 22255 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab26adcde329419380eb61b4e02881bc" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43721 } }
I20260928 13:11:03.693221 22254 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab26adcde329419380eb61b4e02881bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab26adcde329419380eb61b4e02881bc" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43721 } } }
I20260928 13:11:03.693256 22256 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab26adcde329419380eb61b4e02881bc. Latest consensus state: current_term: 1 leader_uuid: "ab26adcde329419380eb61b4e02881bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab26adcde329419380eb61b4e02881bc" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 43721 } } }
I20260928 13:11:03.693382 22254 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:03.693393 22256 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:03.693740 22261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:11:03.694311 22261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 13:11:03 dist-test-slave-p1lc krb5kdc[22168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:03.696009 22261 catalog_manager.cc:1347] Generated new cluster ID: 5bb2376abf1b42e59a68e6ac6c70f60f
I20260928 13:11:03.696069 22261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:11:03.699370 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:03.687446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49764 (local address 127.8.88.190:43721)
0928 13:11:03.687651 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:03.687661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:03.687696 (+ 35us) server_negotiation.cc:407] Connection header received
0928 13:11:03.688286 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:03.688295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:03.688358 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:03.688533 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:03.689235 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:03.690713 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:03.691728 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:03.692365 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:03.692486 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:03.695562 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:03.695566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:03.695566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:03.695578 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:03.697635 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:03.698097 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:03.698100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:03.698100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:03.698204 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:03.698440 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:03.698441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:03.698441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:03.698730 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:03.698881 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:03.699010 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 13:11:03.699143 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
I20260928 13:11:03.700656 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:11:03.706537 22261 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:11:03.707145 22261 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:11:03.712529 22261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab26adcde329419380eb61b4e02881bc: Generated new TSK 0
I20260928 13:11:03.712770 22261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:11:03.754812 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 22181
2026-09-28T13:11:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1366 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:11:03 dist-test-slave-p1lc krb5kdc[22276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:11:03 dist-test-slave-p1lc krb5kdc[22276](info): set up 2 sockets
Sep 28 13:11:03 dist-test-slave-p1lc krb5kdc[22276](info): commencing operation
krb5kdc: starting...
W20260928 13:11:04.794112 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:11:04 dist-test-slave-p1lc krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:11:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:11:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:11:04.924685 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46775
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:05.009958 22289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:05.010233 22289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:05.010277 22289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:05.011654 22289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:11:05.011745 22289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:05.011772 22289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:05.011790 22289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:11:05.011811 22289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:11:05.013468 22289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:05.013747 22289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:05.013993 22289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:05.019361 22295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:05.019431 22289 server_base.cc:1034] running on GCE node
W20260928 13:11:05.019362 22297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:05.019362 22294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:05.019870 22289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:05.020177 22289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:05.021355 22289 hybrid_clock.cc:648] HybridClock initialized: now 1790601065021307 us; error 68 us; skew 500 ppm
Sep 28 13:11:05 dist-test-slave-p1lc krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601065, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:05.023926 22289 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:11:05.024413 22289 webserver.cc:460] Webserver started at http://127.8.88.190:35669/ using document root <none> and password file <none>
I20260928 13:11:05.024699 22289 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:05.024747 22289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:05.024950 22289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:11:05.026037 22289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "889d186917ff4470ad633a821f50d51e"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "c73eb95194d98caf9cd67ad66a6c5af2"
server_key_iv: "f041cf8101c58271d83d7f9c039fec2a"
server_key_version: "encryptionkey@0"
I20260928 13:11:05.026405 22289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "889d186917ff4470ad633a821f50d51e"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "c73eb95194d98caf9cd67ad66a6c5af2"
server_key_iv: "f041cf8101c58271d83d7f9c039fec2a"
server_key_version: "encryptionkey@0"
I20260928 13:11:05.027743 22289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:11:05.028606 22304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:05.028800 22289 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:05.028927 22289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "889d186917ff4470ad633a821f50d51e"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "c73eb95194d98caf9cd67ad66a6c5af2"
server_key_iv: "f041cf8101c58271d83d7f9c039fec2a"
server_key_version: "encryptionkey@0"
I20260928 13:11:05.029017 22289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:05.051470 22289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:05.053644 22289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:05.053841 22289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:05.058051 22289 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:46775
I20260928 13:11:05.058087 22356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:46775 every 8 connection(s)
I20260928 13:11:05.058454 22289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:11:05.059526 22357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:05.060067 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 22289
I20260928 13:11:05.060179 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:11:05.060407 8546 external_mini_cluster.cc:1428] Setting key ed14937bbef3a685b6fc50fc404670d8
I20260928 13:11:05.063009 22357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Bootstrap starting.
I20260928 13:11:05.063752 22357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:05.064078 22357 log.cc:824] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:05.064836 22357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: No bootstrap required, opened a new log
I20260928 13:11:05.067207 22357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } }
I20260928 13:11:05.067395 22357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:05.067446 22357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889d186917ff4470ad633a821f50d51e, State: Initialized, Role: FOLLOWER
I20260928 13:11:05.067611 22357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } }
I20260928 13:11:05.067757 22357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:11:05.067809 22357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:11:05.067871 22357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 0 FOLLOWER]: Advancing to term 1
Sep 28 13:11:05 dist-test-slave-p1lc krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:05.068652 22357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } }
I20260928 13:11:05.068782 22357 leader_election.cc:302] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 889d186917ff4470ad633a821f50d51e; no voters:
I20260928 13:11:05.069020 22357 leader_election.cc:288] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:11:05.069078 22362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:11:05.069262 22362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 1 LEADER]: Becoming Leader. State: Replica: 889d186917ff4470ad633a821f50d51e, State: Running, Role: LEADER
I20260928 13:11:05.069350 22357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:11:05.069383 22362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } }
I20260928 13:11:05.069864 22363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "889d186917ff4470ad633a821f50d51e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } } }
I20260928 13:11:05.069957 22363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:05.070338 22364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 889d186917ff4470ad633a821f50d51e. Latest consensus state: current_term: 1 leader_uuid: "889d186917ff4470ad633a821f50d51e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } } }
I20260928 13:11:05.070408 22364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:05.070523 22369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:11:05.071034 22369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:11:05.072741 22369 catalog_manager.cc:1347] Generated new cluster ID: 78a93f0dfcc94fc4a2d672394ce81067
I20260928 13:11:05.072791 22369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:11:05.073392 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.060992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.8.88.190:46775)
0928 13:11:05.061181 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:05.061189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:05.061215 (+ 26us) server_negotiation.cc:407] Connection header received
0928 13:11:05.061615 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:05.061621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:05.061666 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:05.061820 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:05.062744 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.064238 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.065162 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.065910 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.066040 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.068502 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:05.068506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:05.068506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:05.068516 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:05.070892 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:05.071437 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:05.071440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:05.071441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:05.071538 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:05.072122 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:05.072123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:05.072123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:05.072462 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:05.073051 (+ 589us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:05.073126 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 13:11:05.073236 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260928 13:11:05.109558 22369 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:11:05.110162 22369 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:11:05.116379 22369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Generated new TSK 0
I20260928 13:11:05.116585 22369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:11:05.151648 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46775
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:05.246409 22383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:05.246624 22383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:05.246649 22383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:05.247920 22383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:05.247970 22383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:05.247998 22383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:11:05.249526 22383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:05.249791 22383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:05.250061 22383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:05.255275 22391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:05.255252 22389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:05.255256 22388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:05.255501 22383 server_base.cc:1034] running on GCE node
I20260928 13:11:05.255738 22383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:05.255966 22383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:05.257139 22383 hybrid_clock.cc:648] HybridClock initialized: now 1790601065257115 us; error 34 us; skew 500 ppm
Sep 28 13:11:05 dist-test-slave-p1lc krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601065, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:05.259976 22383 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:11:05.260552 22383 webserver.cc:460] Webserver started at http://127.8.88.129:39057/ using document root <none> and password file <none>
I20260928 13:11:05.260807 22383 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:05.260913 22383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:05.261070 22383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:11:05.262169 22383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "1ffea481b6204c6b824aa394070b6f79"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "dd85bb84f2a960650cbd4c29224025dd"
server_key_iv: "b9f3953cbd5d12bbd3e3611cc21701f2"
server_key_version: "encryptionkey@0"
I20260928 13:11:05.262537 22383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "1ffea481b6204c6b824aa394070b6f79"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "dd85bb84f2a960650cbd4c29224025dd"
server_key_iv: "b9f3953cbd5d12bbd3e3611cc21701f2"
server_key_version: "encryptionkey@0"
I20260928 13:11:05.264008 22383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:11:05.264942 22398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:05.265120 22383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:11:05.265244 22383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "1ffea481b6204c6b824aa394070b6f79"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "dd85bb84f2a960650cbd4c29224025dd"
server_key_iv: "b9f3953cbd5d12bbd3e3611cc21701f2"
server_key_version: "encryptionkey@0"
I20260928 13:11:05.265326 22383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:05.274165 22383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:05.276372 22383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:05.276554 22383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:05.276829 22383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:05.277226 22383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:11:05.277285 22383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:05.277328 22383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:11:05.277355 22383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:05.283581 22383 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:40083
I20260928 13:11:05.283627 22511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:40083 every 8 connection(s)
I20260928 13:11:05.283975 22383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:11:05.287712 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 22383
I20260928 13:11:05.287798 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:11:05.288023 8546 external_mini_cluster.cc:1428] Setting key f7af91aed8834a4f26976603086a0ff7
Sep 28 13:11:05 dist-test-slave-p1lc krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601065, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:05.294436 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.284808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:43599 (local address 127.8.88.190:46775)
0928 13:11:05.284909 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:05.284911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:05.285197 (+ 286us) server_negotiation.cc:407] Connection header received
0928 13:11:05.285750 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:05.285751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:05.285763 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:05.285809 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:05.286847 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.287367 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.288601 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.288926 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.288990 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.291507 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:05.291508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:05.291508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:05.291515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:05.293013 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:05.293450 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:05.293451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:05.293451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:05.293512 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:05.293762 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:05.293763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:05.293763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:05.293893 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:05.293972 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:05.294250 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 13:11:05.294340 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 13:11:05.294646 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.284778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:43599)
0928 13:11:05.285040 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:05.285084 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:05.285639 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:05.285879 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:05.285885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:05.286533 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:05.286732 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.286737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.287469 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.287470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.288517 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.288519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.288998 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.288999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.289118 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.289685 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:05.289693 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:05.291407 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:05.293111 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:05.293119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:05.293121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:05.293349 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:05.293580 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:05.293581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:05.293581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:05.293692 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:05.293962 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:05.293969 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:05.294116 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 13:11:05.294229 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 13:11:05.294404 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":119,"threads_started":1}
I20260928 13:11:05.295179 22512 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46775
I20260928 13:11:05.295290 22512 heartbeater.cc:464] Registering TS with master...
I20260928 13:11:05.295514 22512 heartbeater.cc:511] Master 127.8.88.190:46775 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
I20260928 13:11:05.296113 22321 ts_manager.cc:194] Registered new tserver with Master: 1ffea481b6204c6b824aa394070b6f79 (127.8.88.129:40083)
I20260928 13:11:05.296830 22321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:43599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:11:05.317261 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46775
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:05.398384 22517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:05.398591 22517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:05.398615 22517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:05.399852 22517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:05.399904 22517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:05.399930 22517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:11:05.401415 22517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:05.401659 22517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:05.401896 22517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:05.406877 22525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:05.406908 22522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:05.406971 22523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:05.407087 22517 server_base.cc:1034] running on GCE node
I20260928 13:11:05.407289 22517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:05.407536 22517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:05.408706 22517 hybrid_clock.cc:648] HybridClock initialized: now 1790601065408684 us; error 39 us; skew 500 ppm
Sep 28 13:11:05 dist-test-slave-p1lc krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601065, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:05.411391 22517 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:11:05.411907 22517 webserver.cc:460] Webserver started at http://127.8.88.130:34267/ using document root <none> and password file <none>
I20260928 13:11:05.412151 22517 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:05.412220 22517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:05.412376 22517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:11:05.413475 22517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "7a662878cdc4402181d95cb0bf679a2b"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "b899baa509aba22014acb1fa3d454d36"
server_key_iv: "76b9828ece31024832d0f6522c6132ee"
server_key_version: "encryptionkey@0"
I20260928 13:11:05.413825 22517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "7a662878cdc4402181d95cb0bf679a2b"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "b899baa509aba22014acb1fa3d454d36"
server_key_iv: "76b9828ece31024832d0f6522c6132ee"
server_key_version: "encryptionkey@0"
I20260928 13:11:05.415129 22517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 13:11:05.416026 22532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:05.416209 22517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:05.416309 22517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "7a662878cdc4402181d95cb0bf679a2b"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "b899baa509aba22014acb1fa3d454d36"
server_key_iv: "76b9828ece31024832d0f6522c6132ee"
server_key_version: "encryptionkey@0"
I20260928 13:11:05.416380 22517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:05.424741 22517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:05.427217 22517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:05.427386 22517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:05.427614 22517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:05.427930 22517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:11:05.427985 22517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:05.428027 22517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:11:05.428056 22517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:05.434156 22517 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:38115
I20260928 13:11:05.434195 22645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:38115 every 8 connection(s)
I20260928 13:11:05.434536 22517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
Sep 28 13:11:05 dist-test-slave-p1lc krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601065, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:05.442134 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 22517
I20260928 13:11:05.442271 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:11:05.442469 8546 external_mini_cluster.cc:1428] Setting key 92b3908f2381880a3e869bd0176f671c
I20260928 13:11:05.444674 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.435254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:55021 (local address 127.8.88.190:46775)
0928 13:11:05.435297 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:05.435299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:05.435540 (+ 241us) server_negotiation.cc:407] Connection header received
0928 13:11:05.436001 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:05.436002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:05.436012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:05.436065 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:05.437106 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.437677 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.438737 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.438933 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.438990 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.441397 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:05.441408 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:05.441409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:05.441418 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:05.443217 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:05.443627 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:05.443628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:05.443629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:05.443687 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:05.443964 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:05.443965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:05.443966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:05.444087 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:05.444158 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:05.444518 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 13:11:05.444610 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 13:11:05.444972 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.435256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:55021)
0928 13:11:05.435408 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:05.435443 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:05.435921 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:05.436120 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:05.436123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:05.436687 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:05.436994 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.437001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.437779 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.437780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.438649 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.438653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.439010 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.439011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.439117 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.439734 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:05.439743 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:05.441254 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:05.443300 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:05.443307 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:05.443310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:05.443549 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:05.443766 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:05.443767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:05.443767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:05.443881 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:05.444163 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:05.444169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:05.444333 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 13:11:05.444462 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 13:11:05.444661 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260928 13:11:05.445454 22646 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46775
I20260928 13:11:05.445571 22646 heartbeater.cc:464] Registering TS with master...
I20260928 13:11:05.445760 22646 heartbeater.cc:511] Master 127.8.88.190:46775 requested a full tablet report, sending...
I20260928 13:11:05.446170 22321 ts_manager.cc:194] Registered new tserver with Master: 7a662878cdc4402181d95cb0bf679a2b (127.8.88.130:38115)
I20260928 13:11:05.446570 22321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:55021
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:11:05.472534 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46775
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:05.557066 22651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:05.557300 22651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:05.557330 22651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:05.558657 22651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:05.558720 22651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:05.558750 22651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:11:05.560443 22651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:05.560750 22651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:05.561069 22651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:05.567003 22657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:05.567009 22659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:05.566969 22656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:05.567189 22651 server_base.cc:1034] running on GCE node
I20260928 13:11:05.567726 22651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:05.568059 22651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:05.569276 22651 hybrid_clock.cc:648] HybridClock initialized: now 1790601065569265 us; error 42 us; skew 500 ppm
Sep 28 13:11:05 dist-test-slave-p1lc krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601065, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:05.572273 22651 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:11:05.572780 22651 webserver.cc:460] Webserver started at http://127.8.88.131:36171/ using document root <none> and password file <none>
I20260928 13:11:05.573063 22651 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:05.573114 22651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:05.573249 22651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:11:05.574345 22651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "9963837be23a4215a24c6ac251f5f11a"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "e07e143e99eccfb2b1ac43865ea9a46c"
server_key_iv: "daad6d02b782d9f24305a853325a048a"
server_key_version: "encryptionkey@0"
I20260928 13:11:05.574738 22651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "9963837be23a4215a24c6ac251f5f11a"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "e07e143e99eccfb2b1ac43865ea9a46c"
server_key_iv: "daad6d02b782d9f24305a853325a048a"
server_key_version: "encryptionkey@0"
I20260928 13:11:05.576185 22651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:11:05.577142 22666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:05.577320 22651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:05.577427 22651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "9963837be23a4215a24c6ac251f5f11a"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "e07e143e99eccfb2b1ac43865ea9a46c"
server_key_iv: "daad6d02b782d9f24305a853325a048a"
server_key_version: "encryptionkey@0"
I20260928 13:11:05.577510 22651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:05.600494 22651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:05.603283 22651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:05.603466 22651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:05.603731 22651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:05.604084 22651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:11:05.604138 22651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:05.604182 22651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:11:05.604211 22651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:05.610760 22651 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:42865
I20260928 13:11:05.610806 22779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:42865 every 8 connection(s)
I20260928 13:11:05.611179 22651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
Sep 28 13:11:05 dist-test-slave-p1lc krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601065, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:05.618557 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 22651
I20260928 13:11:05.618672 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:11:05.618932 8546 external_mini_cluster.cc:1428] Setting key ca543e14b3c6e5989b8669ac74838e46
I20260928 13:11:05.621656 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.611997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:35767 (local address 127.8.88.190:46775)
0928 13:11:05.612065 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:05.612067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:05.612244 (+ 177us) server_negotiation.cc:407] Connection header received
0928 13:11:05.612795 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:05.612796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:05.612808 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:05.612955 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:05.613948 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.614518 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.615580 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.615774 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.615834 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.618348 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:05.618350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:05.618351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:05.618360 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:05.620190 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:05.620593 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:05.620595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:05.620595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:05.620655 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:05.620978 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:05.620980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:05.620980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:05.621109 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:05.621188 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:05.621477 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 13:11:05.621561 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 13:11:05.621896 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.611943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:35767)
0928 13:11:05.612102 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:05.612167 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:05.612667 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:05.613013 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:05.613016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:05.613604 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:05.613818 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.613824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.614636 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.614637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.615490 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.615492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.615877 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.615879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.616062 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.616637 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:05.616646 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:05.618132 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:05.620254 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:05.620261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:05.620264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:05.620501 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:05.620731 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:05.620732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:05.620733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:05.620847 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:05.621177 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:05.621184 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:05.621333 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 13:11:05.621446 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 13:11:05.621623 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260928 13:11:05.622498 22780 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46775
I20260928 13:11:05.622605 22780 heartbeater.cc:464] Registering TS with master...
I20260928 13:11:05.622753 22780 heartbeater.cc:511] Master 127.8.88.190:46775 requested a full tablet report, sending...
I20260928 13:11:05.623170 22321 ts_manager.cc:194] Registered new tserver with Master: 9963837be23a4215a24c6ac251f5f11a (127.8.88.131:42865)
I20260928 13:11:05.623634 22321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:35767
I20260928 13:11:05.631325 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:11:05.639793 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.632745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.8.88.190:46775)
0928 13:11:05.632831 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:05.632834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:05.633011 (+ 177us) server_negotiation.cc:407] Connection header received
0928 13:11:05.633126 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:05.633127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:05.633143 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:05.633205 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:05.633802 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.634305 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.635052 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.635245 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.635322 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.636588 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:05.636590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:05.636590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:05.636600 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:05.638492 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:05.638883 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:05.638884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:05.638885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:05.638950 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:05.639160 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:05.639161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:05.639161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:05.639340 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:05.639487 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:05.639581 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 13:11:05.639672 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 13:11:05.642002 22321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:11:05.643800 22321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:11:05.650817 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.646573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:38115 (local address 127.0.0.1:45598)
0928 13:11:05.646904 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:05.646948 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:05.647026 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:05.647361 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:05.647364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:05.647366 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:05.647441 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.647443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.648815 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.648818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.649776 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.649778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.650477 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.650478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.650557 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.650572 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 13:11:05.650672 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 13:11:05.650729 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":77,"threads_started":1}
I20260928 13:11:05.650821 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.646742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.8.88.130:38115)
0928 13:11:05.646939 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:05.646945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:05.647118 (+ 173us) server_negotiation.cc:407] Connection header received
0928 13:11:05.647166 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:05.647169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:05.647183 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:05.647256 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:05.647525 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.648725 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.649870 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.650383 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.650476 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.650541 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:05.650692 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 13:11:05.650734 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
I20260928 13:11:05.651110 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.646573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:42865 (local address 127.0.0.1:48232)
0928 13:11:05.646922 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:05.646986 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:05.647055 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:05.647385 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:05.647386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:05.647389 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:05.647505 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.647507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.648471 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.648473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.649795 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.649798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.650801 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.650801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.650895 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.650899 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:11:05.650952 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 13:11:05.651012 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":26,"thread_start_us":64,"threads_started":1}
I20260928 13:11:05.651396 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.646943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55386 (local address 127.8.88.129:40083)
0928 13:11:05.647129 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:05.647135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:05.647243 (+ 108us) server_negotiation.cc:407] Connection header received
0928 13:11:05.647281 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:05.647283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:05.647297 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:05.647365 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:05.647562 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.648661 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.649936 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.650649 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.650707 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.650760 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:05.651273 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 13:11:05.651320 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":79,"threads_started":1}
I20260928 13:11:05.651417 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.646713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.8.88.131:42865)
0928 13:11:05.647034 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:05.647038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:05.647056 (+ 18us) server_negotiation.cc:407] Connection header received
0928 13:11:05.647127 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:05.647129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:05.647149 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:05.647306 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:05.647649 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.648355 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.649900 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.650622 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.650701 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.650760 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:05.651298 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 13:11:05.651340 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":111,"threads_started":1}
I20260928 13:11:05.651561 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.646780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:40083 (local address 127.0.0.1:55386)
0928 13:11:05.647161 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:05.647171 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:05.647220 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:05.647437 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:05.647438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:05.647439 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:05.647489 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.647490 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.648770 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.648771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.649850 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.649852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.650799 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.650800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.651053 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.651057 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:11:05.651103 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 13:11:05.651146 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260928 13:11:05.651608 22580 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe2a65ad0e14a55951913d56252393f (DEFAULT_TABLE table=test-table [id=10a5a3a4f80b4e019b43bd1f7c4e0798]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:11:05.652029 22580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe2a65ad0e14a55951913d56252393f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:05.652118 22714 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe2a65ad0e14a55951913d56252393f (DEFAULT_TABLE table=test-table [id=10a5a3a4f80b4e019b43bd1f7c4e0798]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:11:05.652464 22714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe2a65ad0e14a55951913d56252393f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:05.652491 22446 tablet_service.cc:1467] Processing CreateTablet for tablet 0fe2a65ad0e14a55951913d56252393f (DEFAULT_TABLE table=test-table [id=10a5a3a4f80b4e019b43bd1f7c4e0798]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:11:05.652799 22446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fe2a65ad0e14a55951913d56252393f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:05.655224 22797 tablet_bootstrap.cc:492] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b: Bootstrap starting.
I20260928 13:11:05.655797 22798 tablet_bootstrap.cc:492] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a: Bootstrap starting.
I20260928 13:11:05.655882 22797 tablet_bootstrap.cc:654] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:05.656021 22799 tablet_bootstrap.cc:492] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Bootstrap starting.
I20260928 13:11:05.656580 22799 tablet_bootstrap.cc:654] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:05.656618 22797 log.cc:824] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:05.656693 22798 tablet_bootstrap.cc:654] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:05.657234 22799 log.cc:824] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:05.657249 22798 log.cc:824] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:05.657539 22797 tablet_bootstrap.cc:492] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b: No bootstrap required, opened a new log
I20260928 13:11:05.657617 22797 ts_tablet_manager.cc:1397] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:11:05.657995 22799 tablet_bootstrap.cc:492] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: No bootstrap required, opened a new log
I20260928 13:11:05.658007 22798 tablet_bootstrap.cc:492] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a: No bootstrap required, opened a new log
I20260928 13:11:05.658070 22798 ts_tablet_manager.cc:1397] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:11:05.658070 22799 ts_tablet_manager.cc:1397] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:11:05.659266 22797 raft_consensus.cc:348] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } }
I20260928 13:11:05.659440 22797 raft_consensus.cc:374] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:05.659484 22797 raft_consensus.cc:729] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a662878cdc4402181d95cb0bf679a2b, State: Initialized, Role: FOLLOWER
I20260928 13:11:05.659634 22797 consensus_queue.cc:260] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } }
I20260928 13:11:05.659902 22797 ts_tablet_manager.cc:1428] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:11:05.660034 22646 heartbeater.cc:503] Master 127.8.88.190:46775 was elected leader, sending a full tablet report...
I20260928 13:11:05.660010 22798 raft_consensus.cc:348] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } }
I20260928 13:11:05.660010 22799 raft_consensus.cc:348] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } }
I20260928 13:11:05.660190 22799 raft_consensus.cc:374] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:05.660190 22798 raft_consensus.cc:374] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:05.660234 22799 raft_consensus.cc:729] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffea481b6204c6b824aa394070b6f79, State: Initialized, Role: FOLLOWER
I20260928 13:11:05.660234 22798 raft_consensus.cc:729] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9963837be23a4215a24c6ac251f5f11a, State: Initialized, Role: FOLLOWER
I20260928 13:11:05.660354 22799 consensus_queue.cc:260] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } }
I20260928 13:11:05.660354 22798 consensus_queue.cc:260] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } }
I20260928 13:11:05.660632 22798 ts_tablet_manager.cc:1428] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:11:05.660632 22799 ts_tablet_manager.cc:1428] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:11:05.660775 22512 heartbeater.cc:503] Master 127.8.88.190:46775 was elected leader, sending a full tablet report...
I20260928 13:11:05.660789 22780 heartbeater.cc:503] Master 127.8.88.190:46775 was elected leader, sending a full tablet report...
W20260928 13:11:05.685153 22647 tablet.cc:2369] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:11:05.766039 22804 raft_consensus.cc:482] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:11:05.766368 22804 raft_consensus.cc:504] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } }
I20260928 13:11:05.766764 22804 leader_election.cc:288] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a662878cdc4402181d95cb0bf679a2b (127.8.88.130:38115), 1ffea481b6204c6b824aa394070b6f79 (127.8.88.129:40083)
I20260928 13:11:05.770978 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.767096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:46477 (local address 127.8.88.130:38115)
0928 13:11:05.767148 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:05.767151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:05.767162 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:11:05.767254 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:05.767256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:05.767270 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:05.767333 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:05.767497 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.768281 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.769867 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.770527 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.770610 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.770619 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:05.770853 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 13:11:05.770906 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 13:11:05.770978 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.767229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:35867 (local address 127.8.88.129:40083)
0928 13:11:05.767283 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:05.767285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:05.767506 (+ 221us) server_negotiation.cc:407] Connection header received
0928 13:11:05.767557 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:05.767558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:05.767570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:05.767635 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:05.767808 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.768471 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.769873 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.770527 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.770610 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.770619 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:05.770861 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 13:11:05.770906 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 13:11:05.770967 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.766996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:38115 (local address 127.8.88.131:46477)
0928 13:11:05.767067 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:05.767087 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:05.767159 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:05.767338 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:05.767339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:05.767341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:05.767418 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.767420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.768362 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.768363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.769708 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.769712 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.770622 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.770624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.770717 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.770720 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:11:05.770787 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 13:11:05.770836 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":2}
I20260928 13:11:05.770973 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.767114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:40083 (local address 127.8.88.131:35867)
0928 13:11:05.767298 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:05.767308 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:05.767459 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:05.767657 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:05.767657 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:05.767659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:05.767726 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.767728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.768550 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.768551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.769721 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:05.769723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.770623 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.770624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:05.770714 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.770720 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 13:11:05.770787 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 13:11:05.770832 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":4,"thread_start_us":106,"threads_started":1}
I20260928 13:11:05.771466 22600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe2a65ad0e14a55951913d56252393f" candidate_uuid: "9963837be23a4215a24c6ac251f5f11a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a662878cdc4402181d95cb0bf679a2b" is_pre_election: true
I20260928 13:11:05.771620 22466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe2a65ad0e14a55951913d56252393f" candidate_uuid: "9963837be23a4215a24c6ac251f5f11a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ffea481b6204c6b824aa394070b6f79" is_pre_election: true
I20260928 13:11:05.771680 22600 raft_consensus.cc:2454] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9963837be23a4215a24c6ac251f5f11a in term 0.
I20260928 13:11:05.771764 22466 raft_consensus.cc:2454] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9963837be23a4215a24c6ac251f5f11a in term 0.
I20260928 13:11:05.771939 22669 leader_election.cc:302] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a662878cdc4402181d95cb0bf679a2b, 9963837be23a4215a24c6ac251f5f11a; no voters:
I20260928 13:11:05.772096 22804 raft_consensus.cc:2781] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:11:05.772199 22804 raft_consensus.cc:482] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:11:05.772243 22804 raft_consensus.cc:3037] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:05.773056 22804 raft_consensus.cc:504] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } }
I20260928 13:11:05.773218 22804 leader_election.cc:288] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [CANDIDATE]: Term 1 election: Requested vote from peers 7a662878cdc4402181d95cb0bf679a2b (127.8.88.130:38115), 1ffea481b6204c6b824aa394070b6f79 (127.8.88.129:40083)
I20260928 13:11:05.773461 22466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe2a65ad0e14a55951913d56252393f" candidate_uuid: "9963837be23a4215a24c6ac251f5f11a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ffea481b6204c6b824aa394070b6f79"
I20260928 13:11:05.773528 22600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fe2a65ad0e14a55951913d56252393f" candidate_uuid: "9963837be23a4215a24c6ac251f5f11a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a662878cdc4402181d95cb0bf679a2b"
I20260928 13:11:05.773574 22466 raft_consensus.cc:3037] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:05.773662 22600 raft_consensus.cc:3037] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:05.774408 22466 raft_consensus.cc:2454] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9963837be23a4215a24c6ac251f5f11a in term 1.
I20260928 13:11:05.774591 22669 leader_election.cc:302] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ffea481b6204c6b824aa394070b6f79, 9963837be23a4215a24c6ac251f5f11a; no voters:
I20260928 13:11:05.774658 22600 raft_consensus.cc:2454] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9963837be23a4215a24c6ac251f5f11a in term 1.
I20260928 13:11:05.774698 22804 raft_consensus.cc:2781] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:11:05.774955 22804 raft_consensus.cc:686] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [term 1 LEADER]: Becoming Leader. State: Replica: 9963837be23a4215a24c6ac251f5f11a, State: Running, Role: LEADER
I20260928 13:11:05.775084 22804 consensus_queue.cc:237] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } }
I20260928 13:11:05.776134 22321 catalog_manager.cc:5495] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9963837be23a4215a24c6ac251f5f11a (127.8.88.131). New cstate: current_term: 1 leader_uuid: "9963837be23a4215a24c6ac251f5f11a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } health_report { overall_health: HEALTHY } } }
W20260928 13:11:05.784821 22513 tablet.cc:2369] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:11:05.798462 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.795772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48236 (local address 127.8.88.131:42865)
0928 13:11:05.795848 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:05.795850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:05.795884 (+ 34us) server_negotiation.cc:407] Connection header received
0928 13:11:05.795956 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:05.795957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:05.795978 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:05.796071 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:11:05.796244 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.797002 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.797722 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:05.797913 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:05.797971 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:05.798102 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:11:05.798207 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:11:05.798265 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:05.798299 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 13:11:05.798315 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 13:11:05.800495 22466 raft_consensus.cc:1261] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Refusing update from remote peer 9963837be23a4215a24c6ac251f5f11a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:11:05.800484 22600 raft_consensus.cc:1261] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b [term 1 FOLLOWER]: Refusing update from remote peer 9963837be23a4215a24c6ac251f5f11a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:11:05.801002 22807 consensus_queue.cc:1035] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:11:05.801164 22804 consensus_queue.cc:1035] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:11:05.805987 22321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:11:05.806085 22321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:11:05.806821 22321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Sending DeleteTablet for 3 replicas of tablet 0fe2a65ad0e14a55951913d56252393f
I20260928 13:11:05.807209 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 22383
I20260928 13:11:05.807215 22580 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe2a65ad0e14a55951913d56252393f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:11:05 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:45598
I20260928 13:11:05.807211 22446 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe2a65ad0e14a55951913d56252393f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:11:05 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:55386
I20260928 13:11:05.807976 22714 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe2a65ad0e14a55951913d56252393f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:11:05 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:48232
I20260928 13:11:05.808202 22821 tablet_replica.cc:330] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a: stopping tablet replica
I20260928 13:11:05.808225 22820 tablet_replica.cc:330] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b: stopping tablet replica
I20260928 13:11:05.808302 22821 raft_consensus.cc:2229] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:11:05.808321 22820 raft_consensus.cc:2229] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:11:05.808390 22821 raft_consensus.cc:2258] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:11:05.808421 22820 raft_consensus.cc:2258] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:11:05.808833 22821 ts_tablet_manager.cc:1905] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:11:05.811161 22820 ts_tablet_manager.cc:1905] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:11:05.811201 22821 ts_tablet_manager.cc:1918] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:11:05.811275 22821 log.cc:1196] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/0fe2a65ad0e14a55951913d56252393f
I20260928 13:11:05.811625 22821 ts_tablet_manager.cc:1939] T 0fe2a65ad0e14a55951913d56252393f P 9963837be23a4215a24c6ac251f5f11a: Deleting consensus metadata
I20260928 13:11:05.812252 22306 catalog_manager.cc:4841] TS 9963837be23a4215a24c6ac251f5f11a (127.8.88.131:42865): tablet 0fe2a65ad0e14a55951913d56252393f (table test-table [id=10a5a3a4f80b4e019b43bd1f7c4e0798]) successfully deleted
I20260928 13:11:05.813727 22820 ts_tablet_manager.cc:1918] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:11:05.813817 22820 log.cc:1196] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/0fe2a65ad0e14a55951913d56252393f
W20260928 13:11:05.813900 22307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:40083 (error 108)
I20260928 13:11:05.814090 22820 ts_tablet_manager.cc:1939] T 0fe2a65ad0e14a55951913d56252393f P 7a662878cdc4402181d95cb0bf679a2b: Deleting consensus metadata
I20260928 13:11:05.814181 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 22517
I20260928 13:11:05.819137 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 22651
I20260928 13:11:05.819201 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.819013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:40083 (local address 127.0.0.1:55400)
0928 13:11:05.819107 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:05.819159 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:40083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 13:11:05.819288 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:05.819103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:38115 (local address 127.0.0.1:45614)
0928 13:11:05.819162 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:05.819196 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.130:38115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 13:11:05.819949 22307 catalog_manager.cc:4584] TS 1ffea481b6204c6b824aa394070b6f79 (127.8.88.129:40083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0fe2a65ad0e14a55951913d56252393f: Network error: Client connection negotiation failed: client connection to 127.8.88.129:40083: connect: Connection refused (error 111)
I20260928 13:11:05.824074 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 22289
I20260928 13:11:05.829933 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46775
--webserver_interface=127.8.88.190
--webserver_port=35669
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:05.913307 22823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:05.913583 22823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:05.913625 22823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:05.914939 22823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:11:05.915019 22823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:05.915046 22823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:05.915081 22823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:11:05.915107 22823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:11:05.916811 22823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=35669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:05.917160 22823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:05.917421 22823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:05.926929 22829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:05.926994 22823 server_base.cc:1034] running on GCE node
W20260928 13:11:05.926964 22830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:05.926957 22832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:05.927537 22823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:05.927850 22823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:05.928970 22823 hybrid_clock.cc:648] HybridClock initialized: now 1790601065928951 us; error 28 us; skew 500 ppm
Sep 28 13:11:05 dist-test-slave-p1lc krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601065, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:05.938745 22823 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:11:05.939374 22823 webserver.cc:460] Webserver started at http://127.8.88.190:35669/ using document root <none> and password file <none>
I20260928 13:11:05.939646 22823 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:05.939725 22823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:05.941538 22823 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:11:05.942363 22839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:05.942615 22823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:05.942740 22823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "889d186917ff4470ad633a821f50d51e"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "c73eb95194d98caf9cd67ad66a6c5af2"
server_key_iv: "f041cf8101c58271d83d7f9c039fec2a"
server_key_version: "encryptionkey@0"
I20260928 13:11:05.943215 22823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:05.961613 22823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:05.964041 22823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:05.964234 22823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:05.968704 22823 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:46775
I20260928 13:11:05.968734 22891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:46775 every 8 connection(s)
I20260928 13:11:05.969230 22823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:11:05.971139 22892 sys_catalog.cc:263] Verifying existing consensus state
I20260928 13:11:05.971779 22892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Bootstrap starting.
I20260928 13:11:05.975582 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 22823
I20260928 13:11:05.975924 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:40083
--local_ip_for_outbound_sockets=127.8.88.129
--tserver_master_addrs=127.8.88.190:46775
--webserver_port=39057
--webserver_interface=127.8.88.129
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:11:05.981046 22892 log.cc:824] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:05.983105 22892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:11:05.983460 22892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Bootstrap complete.
I20260928 13:11:05.986369 22892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } }
I20260928 13:11:05.986649 22892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889d186917ff4470ad633a821f50d51e, State: Initialized, Role: FOLLOWER
I20260928 13:11:05.986835 22892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } }
I20260928 13:11:05.986914 22892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:11:05.986975 22892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:11:05.987027 22892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 1 FOLLOWER]: Advancing to term 2
I20260928 13:11:05.988158 22892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } }
I20260928 13:11:05.988253 22892 leader_election.cc:302] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 889d186917ff4470ad633a821f50d51e; no voters:
I20260928 13:11:05.988512 22892 leader_election.cc:288] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 13:11:05.988556 22896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 13:11:05.988718 22896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 2 LEADER]: Becoming Leader. State: Replica: 889d186917ff4470ad633a821f50d51e, State: Running, Role: LEADER
I20260928 13:11:05.988806 22892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:11:05.988935 22896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } }
I20260928 13:11:05.989259 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 889d186917ff4470ad633a821f50d51e. Latest consensus state: current_term: 2 leader_uuid: "889d186917ff4470ad633a821f50d51e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } } }
I20260928 13:11:05.989362 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:05.989261 22897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "889d186917ff4470ad633a821f50d51e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } } }
I20260928 13:11:05.989527 22897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:05.989774 22905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:11:05.990768 22905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:11:05.990919 22905 catalog_manager.cc:1259] Loaded cluster ID: 78a93f0dfcc94fc4a2d672394ce81067
I20260928 13:11:05.991178 22905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:11:05.991487 22905 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 13:11:05.996719 22905 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 13:11:06.000618 22905 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:11:06.000854 22905 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Loaded TSK: 0
I20260928 13:11:06.001142 22905 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 13:11:06.068055 22894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:06.068316 22894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:06.068362 22894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:06.069687 22894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:06.069761 22894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:06.069811 22894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:11:06.071551 22894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:40083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=39057
--tserver_master_addrs=127.8.88.190:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:06.071831 22894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:06.072084 22894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:06.079073 22920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:06.079083 22919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:06.079298 22894 server_base.cc:1034] running on GCE node
W20260928 13:11:06.079463 22922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:06.079686 22894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:06.079983 22894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:06.081164 22894 hybrid_clock.cc:648] HybridClock initialized: now 1790601066081139 us; error 37 us; skew 500 ppm
Sep 28 13:11:06 dist-test-slave-p1lc krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601066, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:06.084178 22894 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:11:06.084764 22894 webserver.cc:460] Webserver started at http://127.8.88.129:39057/ using document root <none> and password file <none>
I20260928 13:11:06.085224 22894 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:06.085438 22894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:06.087097 22894 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:11:06.087921 22929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:06.088142 22894 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:06.088270 22894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "1ffea481b6204c6b824aa394070b6f79"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "dd85bb84f2a960650cbd4c29224025dd"
server_key_iv: "b9f3953cbd5d12bbd3e3611cc21701f2"
server_key_version: "encryptionkey@0"
I20260928 13:11:06.088721 22894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:06.097288 22894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:06.100570 22894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:06.100759 22894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:06.101315 22894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:06.101838 22936 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 13:11:06.103876 22894 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 13:11:06.103967 22894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 13:11:06.104030 22894 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 13:11:06.104792 22894 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 13:11:06.104866 22894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 13:11:06.105028 22936 tablet_bootstrap.cc:492] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Bootstrap starting.
I20260928 13:11:06.113133 22894 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:40083
I20260928 13:11:06.113307 23043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:40083 every 8 connection(s)
I20260928 13:11:06.113648 22894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:11:06.116887 22936 log.cc:824] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:06.120035 22936 tablet_bootstrap.cc:492] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:11:06.120391 22936 tablet_bootstrap.cc:492] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Bootstrap complete.
I20260928 13:11:06.121660 22936 ts_tablet_manager.cc:1397] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.014s
Sep 28 13:11:06 dist-test-slave-p1lc krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601066, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:06.121847 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 22894
I20260928 13:11:06.122066 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:38115
--local_ip_for_outbound_sockets=127.8.88.130
--tserver_master_addrs=127.8.88.190:46775
--webserver_port=34267
--webserver_interface=127.8.88.130
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:11:06.123016 22936 raft_consensus.cc:348] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } }
I20260928 13:11:06.123348 22936 raft_consensus.cc:729] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffea481b6204c6b824aa394070b6f79, State: Initialized, Role: FOLLOWER
I20260928 13:11:06.123490 22936 consensus_queue.cc:260] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } }
I20260928 13:11:06.123777 22936 ts_tablet_manager.cc:1428] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 13:11:06.127687 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.114473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:48181)
0928 13:11:06.114693 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:06.114751 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:06.115237 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:06.115914 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:06.115918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:06.116215 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:06.116419 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.116424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.118105 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.118107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.118961 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.118964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.119732 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.119734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.119885 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.120550 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:06.120559 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:06.123705 (+ 3146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:06.125899 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:06.125908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:06.125910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:06.126155 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:06.126437 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:06.126438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:06.126438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:06.126590 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:06.127003 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:06.127010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:06.127159 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 13:11:06.127276 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 13:11:06.127475 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
I20260928 13:11:06.127712 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.114644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:48181 (local address 127.8.88.190:46775)
0928 13:11:06.115534 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:06.115542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:06.115571 (+ 29us) server_negotiation.cc:407] Connection header received
0928 13:11:06.115677 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:06.115682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:06.115730 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:06.115884 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:06.116573 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.117959 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.119129 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.119599 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.119735 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.123893 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:06.123899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:06.123899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:06.123915 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:06.125771 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:06.126250 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:06.126254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:06.126254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:06.126362 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:06.126666 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:06.126668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:06.126668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:06.126926 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:06.127023 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:06.127336 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 13:11:06.127475 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":85,"threads_started":1}
I20260928 13:11:06.128912 23044 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46775
I20260928 13:11:06.129033 23044 heartbeater.cc:464] Registering TS with master...
I20260928 13:11:06.129218 23044 heartbeater.cc:511] Master 127.8.88.190:46775 requested a full tablet report, sending...
I20260928 13:11:06.129971 22856 ts_manager.cc:194] Registered new tserver with Master: 1ffea481b6204c6b824aa394070b6f79 (127.8.88.129:40083)
I20260928 13:11:06.131260 22856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:48181
I20260928 13:11:06.132170 23044 heartbeater.cc:503] Master 127.8.88.190:46775 was elected leader, sending a full tablet report...
I20260928 13:11:06.136108 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.130907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.8.88.129:40083)
0928 13:11:06.131100 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:06.131105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:06.133098 (+ 1993us) server_negotiation.cc:407] Connection header received
0928 13:11:06.133195 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:06.133198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:06.133212 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:06.133289 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:06.133466 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.134162 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.135185 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.135672 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.135756 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.135815 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:06.135993 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 13:11:06.136032 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":108,"threads_started":1}
I20260928 13:11:06.136108 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.130650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:40083 (local address 127.0.0.1:55402)
0928 13:11:06.132995 (+ 2345us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:06.133027 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:06.133118 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:06.133298 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:06.133301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:06.133302 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:06.133376 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.133379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.134249 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.134251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.135080 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.135083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.135768 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.135769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.135858 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.135870 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 13:11:06.135974 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 13:11:06.136032 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2308,"thread_start_us":83,"threads_started":1}
I20260928 13:11:06.136811 22978 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fe2a65ad0e14a55951913d56252393f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:11:05 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:55402
I20260928 13:11:06.137408 23053 tablet_replica.cc:330] stopping tablet replica
I20260928 13:11:06.137538 23053 raft_consensus.cc:2229] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:11:06.137614 23053 raft_consensus.cc:2258] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:11:06.137926 23053 ts_tablet_manager.cc:1905] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:11:06.139983 23053 ts_tablet_manager.cc:1918] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:11:06.140074 23053 log.cc:1196] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal/wals/0fe2a65ad0e14a55951913d56252393f
I20260928 13:11:06.140348 23053 ts_tablet_manager.cc:1939] T 0fe2a65ad0e14a55951913d56252393f P 1ffea481b6204c6b824aa394070b6f79: Deleting consensus metadata
I20260928 13:11:06.141594 22842 catalog_manager.cc:4841] TS 1ffea481b6204c6b824aa394070b6f79 (127.8.88.129:40083): tablet 0fe2a65ad0e14a55951913d56252393f (table test-table [id=10a5a3a4f80b4e019b43bd1f7c4e0798]) successfully deleted
W20260928 13:11:06.224933 23049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:06.225174 23049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:06.225203 23049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:06.226622 23049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:06.226686 23049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:06.226717 23049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:11:06.228780 23049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:38115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=34267
--tserver_master_addrs=127.8.88.190:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:06.229189 23049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:06.229485 23049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:06.236984 23061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:06.237460 23059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:06.237715 23049 server_base.cc:1034] running on GCE node
W20260928 13:11:06.237632 23058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:06.238106 23049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:06.238503 23049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:06.239722 23049 hybrid_clock.cc:648] HybridClock initialized: now 1790601066239669 us; error 73 us; skew 500 ppm
Sep 28 13:11:06 dist-test-slave-p1lc krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601066, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:06.243284 23049 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:11:06.243979 23049 webserver.cc:460] Webserver started at http://127.8.88.130:34267/ using document root <none> and password file <none>
I20260928 13:11:06.244328 23049 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:06.244433 23049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:06.246258 23049 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:06.247200 23068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:06.247394 23049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:06.247503 23049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "7a662878cdc4402181d95cb0bf679a2b"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "b899baa509aba22014acb1fa3d454d36"
server_key_iv: "76b9828ece31024832d0f6522c6132ee"
server_key_version: "encryptionkey@0"
I20260928 13:11:06.247972 23049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:06.263201 23049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:06.266184 23049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:06.266345 23049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:06.266654 23049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:06.267014 23049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:11:06.267050 23049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:06.267072 23049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:11:06.267086 23049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:06.273680 23049 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:38115
I20260928 13:11:06.273784 23181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:38115 every 8 connection(s)
I20260928 13:11:06.274245 23049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
Sep 28 13:11:06 dist-test-slave-p1lc krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601066, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:06.282544 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 23049
I20260928 13:11:06.282981 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:42865
--local_ip_for_outbound_sockets=127.8.88.131
--tserver_master_addrs=127.8.88.190:46775
--webserver_port=36171
--webserver_interface=127.8.88.131
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:11:06.285705 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.275200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:43225 (local address 127.8.88.190:46775)
0928 13:11:06.275265 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:06.275271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:06.275494 (+ 223us) server_negotiation.cc:407] Connection header received
0928 13:11:06.276025 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:06.276026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:06.276040 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:06.276114 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:06.277251 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.277835 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.278971 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.279180 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.279247 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.281843 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:06.281845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:06.281845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:06.281853 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:06.283657 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:06.284324 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:06.284326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:06.284326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:06.284402 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:06.284726 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:06.284728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:06.284728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:06.284989 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:06.285077 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:06.285463 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 13:11:06.285588 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 13:11:06.285827 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.275102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:43225)
0928 13:11:06.275301 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:06.275353 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:06.275899 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:06.276195 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:06.276199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:06.276855 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:06.277110 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.277115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.277965 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.277966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.278857 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.278859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.279274 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.279275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.279381 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.279987 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:06.279996 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:06.281734 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:06.283754 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:06.283765 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:06.283768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:06.284209 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:06.284504 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:06.284505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:06.284506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:06.284654 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:06.285151 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:06.285158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:06.285298 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 13:11:06.285404 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 13:11:06.285553 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":118,"threads_started":1}
I20260928 13:11:06.286358 23182 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46775
I20260928 13:11:06.286473 23182 heartbeater.cc:464] Registering TS with master...
I20260928 13:11:06.286675 23182 heartbeater.cc:511] Master 127.8.88.190:46775 requested a full tablet report, sending...
I20260928 13:11:06.287159 22856 ts_manager.cc:194] Registered new tserver with Master: 7a662878cdc4402181d95cb0bf679a2b (127.8.88.130:38115)
I20260928 13:11:06.287652 22856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:43225
W20260928 13:11:06.373109 23185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:06.373371 23185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:06.373422 23185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:06.374704 23185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:06.374779 23185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:06.374828 23185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:11:06.376408 23185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:42865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=36171
--tserver_master_addrs=127.8.88.190:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:06.376703 23185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:06.376976 23185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:11:06.382778 23185 server_base.cc:1034] running on GCE node
W20260928 13:11:06.382740 23191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:06.382740 23193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:06.382746 23190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:06.383198 23185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:06.383481 23185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:06.384634 23185 hybrid_clock.cc:648] HybridClock initialized: now 1790601066384606 us; error 39 us; skew 500 ppm
Sep 28 13:11:06 dist-test-slave-p1lc krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601066, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:06.387809 23185 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:11:06.388360 23185 webserver.cc:460] Webserver started at http://127.8.88.131:36171/ using document root <none> and password file <none>
I20260928 13:11:06.388628 23185 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:06.388686 23185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:06.390412 23185 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:06.391139 23200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:06.391415 23185 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:06.391582 23185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "9963837be23a4215a24c6ac251f5f11a"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "e07e143e99eccfb2b1ac43865ea9a46c"
server_key_iv: "daad6d02b782d9f24305a853325a048a"
server_key_version: "encryptionkey@0"
I20260928 13:11:06.391970 23185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:06.431864 23185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:06.434413 23185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:06.434558 23185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:06.434842 23185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:06.435153 23185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:11:06.435189 23185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:06.435212 23185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:11:06.435225 23185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:06.441402 23185 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:42865
I20260928 13:11:06.441438 23313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:42865 every 8 connection(s)
I20260928 13:11:06.441841 23185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
Sep 28 13:11:06 dist-test-slave-p1lc krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601066, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:06.450438 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 23185
I20260928 13:11:06.452356 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.442738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:53887 (local address 127.8.88.190:46775)
0928 13:11:06.442823 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:06.442825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:06.443026 (+ 201us) server_negotiation.cc:407] Connection header received
0928 13:11:06.443545 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:06.443546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:06.443557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:06.443612 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:06.444754 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.445331 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.446489 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.446724 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.446781 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.449147 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:06.449149 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:06.449149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:06.449156 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:06.450763 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:06.451373 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:06.451374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:06.451374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:06.451439 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:06.451676 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:06.451677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:06.451677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:06.451800 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:06.451889 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:06.452174 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 13:11:06.452256 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 13:11:06.452587 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.442705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:53887)
0928 13:11:06.442889 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:06.442927 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:06.443439 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:06.443691 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:06.443695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:06.444422 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:06.444637 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.444643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.445421 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.445422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.446410 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.446413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.446815 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.446816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.446954 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.447483 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:06.447492 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:06.449020 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:06.450906 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:06.450914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:06.450916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:06.451264 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:06.451501 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:06.451501 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:06.451502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:06.451605 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:06.451880 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:06.451887 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:06.452035 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 13:11:06.452152 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 13:11:06.452334 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260928 13:11:06.453171 23314 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46775
I20260928 13:11:06.453296 23314 heartbeater.cc:464] Registering TS with master...
I20260928 13:11:06.453497 23314 heartbeater.cc:511] Master 127.8.88.190:46775 requested a full tablet report, sending...
I20260928 13:11:06.454031 22856 ts_manager.cc:194] Registered new tserver with Master: 9963837be23a4215a24c6ac251f5f11a (127.8.88.131:42865)
I20260928 13:11:06.454478 22856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:53887
I20260928 13:11:06.458953 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.450898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.8.88.190:46775)
0928 13:11:06.450996 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:06.450998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:06.451096 (+ 98us) server_negotiation.cc:407] Connection header received
0928 13:11:06.451142 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:06.451143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:06.451157 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:06.451240 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:06.451921 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.452446 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.453985 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.454269 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.454343 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.455923 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:06.455924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:06.455924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:06.455931 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:06.457671 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:06.458052 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:06.458053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:06.458053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:06.458108 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:06.458409 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:06.458410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:06.458410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:06.458547 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:06.458691 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:06.458745 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 13:11:06.458880 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":48,"threads_started":1}
I20260928 13:11:06.459888 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:11:06.462821 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.460308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.8.88.190:46775)
0928 13:11:06.460429 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:06.460431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:06.460439 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:11:06.460473 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:06.460473 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:06.460506 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:06.460576 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:11:06.460690 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.461266 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.462072 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.462301 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.462363 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.462543 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:11:06.462646 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:11:06.462711 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:06.462743 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 13:11:06.462764 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 13:11:06.463193 22856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:11:06.464917 22856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:11:06.467361 22978 tablet_service.cc:1467] Processing CreateTablet for tablet 69b165f5a6ab4718b29c231acb6dacbe (DEFAULT_TABLE table=test-table [id=1c3b638a3d1b4f8b8927e80b13f141f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:11:06.467581 22978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b165f5a6ab4718b29c231acb6dacbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:06.469285 22936 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: Bootstrap starting.
I20260928 13:11:06.469595 22936 tablet_bootstrap.cc:654] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:06.470170 22936 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: No bootstrap required, opened a new log
I20260928 13:11:06.470222 22936 ts_tablet_manager.cc:1397] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 13:11:06.470345 22936 raft_consensus.cc:348] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:06.470443 22936 raft_consensus.cc:374] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:06.470495 22936 raft_consensus.cc:729] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffea481b6204c6b824aa394070b6f79, State: Initialized, Role: FOLLOWER
I20260928 13:11:06.470541 22936 consensus_queue.cc:260] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:06.470634 22936 ts_tablet_manager.cc:1428] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:06.471323 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.466967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:42865 (local address 127.0.0.1:48246)
0928 13:11:06.467028 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:06.467039 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:06.467078 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:06.467506 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:06.467508 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:06.467509 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:06.467570 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.467571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.468806 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.468807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.469965 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.469967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.471014 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.471015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.471083 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.471088 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:11:06.471121 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 13:11:06.471244 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 13:11:06.471688 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.467088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.8.88.131:42865)
0928 13:11:06.467307 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:06.467311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:06.467321 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:11:06.467353 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:06.467355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:06.467368 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:06.467506 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:06.467799 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.468690 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.470054 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.470906 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.471486 (+ 580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.471562 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:06.471594 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 13:11:06.471628 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":95,"threads_started":1}
I20260928 13:11:06.472381 23248 tablet_service.cc:1467] Processing CreateTablet for tablet 69b165f5a6ab4718b29c231acb6dacbe (DEFAULT_TABLE table=test-table [id=1c3b638a3d1b4f8b8927e80b13f141f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:11:06.472752 23248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b165f5a6ab4718b29c231acb6dacbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:06.473373 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.466967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:38115 (local address 127.0.0.1:45622)
0928 13:11:06.467158 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:06.467167 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:06.467203 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:06.469728 (+ 2525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:06.469729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:06.469731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:06.469768 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.469769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.471029 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.471030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.472155 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.472158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.473150 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.473151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.473205 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.473209 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:11:06.473271 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 13:11:06.473313 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":21,"thread_start_us":60,"threads_started":1}
I20260928 13:11:06.473407 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.467108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.8.88.130:38115)
0928 13:11:06.469496 (+ 2388us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:06.469501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:06.469512 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:11:06.469551 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:06.469553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:06.469569 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:06.469645 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:06.469846 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.470929 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.472237 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.473062 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.473141 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.473197 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:06.473283 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 13:11:06.473327 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2347,"thread_start_us":91,"threads_started":1}
I20260928 13:11:06.474069 23116 tablet_service.cc:1467] Processing CreateTablet for tablet 69b165f5a6ab4718b29c231acb6dacbe (DEFAULT_TABLE table=test-table [id=1c3b638a3d1b4f8b8927e80b13f141f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:11:06.474387 23116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b165f5a6ab4718b29c231acb6dacbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:06.475837 23324 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Bootstrap starting.
I20260928 13:11:06.476498 23324 tablet_bootstrap.cc:654] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:06.477038 23324 log.cc:824] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:06.477245 23325 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: Bootstrap starting.
I20260928 13:11:06.477805 23324 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: No bootstrap required, opened a new log
I20260928 13:11:06.477843 23325 tablet_bootstrap.cc:654] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:06.477900 23324 ts_tablet_manager.cc:1397] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:11:06.478331 23325 log.cc:824] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:06.479203 23325 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: No bootstrap required, opened a new log
I20260928 13:11:06.479285 23325 ts_tablet_manager.cc:1397] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:11:06.479760 23324 raft_consensus.cc:348] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:06.479933 23324 raft_consensus.cc:374] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:06.479978 23324 raft_consensus.cc:729] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9963837be23a4215a24c6ac251f5f11a, State: Initialized, Role: FOLLOWER
I20260928 13:11:06.480094 23324 consensus_queue.cc:260] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:06.480387 23324 ts_tablet_manager.cc:1428] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:11:06.480453 23314 heartbeater.cc:503] Master 127.8.88.190:46775 was elected leader, sending a full tablet report...
I20260928 13:11:06.481078 23325 raft_consensus.cc:348] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:06.481292 23325 raft_consensus.cc:374] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:06.481338 23325 raft_consensus.cc:729] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a662878cdc4402181d95cb0bf679a2b, State: Initialized, Role: FOLLOWER
I20260928 13:11:06.481460 23325 consensus_queue.cc:260] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:06.481782 23182 heartbeater.cc:503] Master 127.8.88.190:46775 was elected leader, sending a full tablet report...
I20260928 13:11:06.482074 23325 ts_tablet_manager.cc:1428] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260928 13:11:06.524986 23183 tablet.cc:2369] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:11:06.582105 23050 raft_consensus.cc:482] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:11:06.582299 23050 raft_consensus.cc:504] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:06.582685 23050 leader_election.cc:288] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a662878cdc4402181d95cb0bf679a2b (127.8.88.130:38115), 9963837be23a4215a24c6ac251f5f11a (127.8.88.131:42865)
I20260928 13:11:06.586017 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.582927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:38115 (local address 127.8.88.129:43451)
0928 13:11:06.583001 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:06.583016 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:06.583086 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:06.583338 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:06.583339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:06.583341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:06.583449 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.583452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.584182 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.584183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.585082 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.585084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.585734 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.585735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.585816 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.585821 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:11:06.585874 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 13:11:06.585959 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 13:11:06.586056 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.583127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:43451 (local address 127.8.88.130:38115)
0928 13:11:06.583203 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:06.583205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:06.583215 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:11:06.583247 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:06.583248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:06.583262 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:06.583319 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:06.583561 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.584097 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.585174 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.585648 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.585710 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.585718 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:06.585919 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 13:11:06.585969 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 13:11:06.586215 23321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.583112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:32915 (local address 127.8.88.131:42865)
0928 13:11:06.583192 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:06.583194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:06.583269 (+ 75us) server_negotiation.cc:407] Connection header received
0928 13:11:06.583339 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:06.583340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:06.583350 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:06.583429 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:06.583674 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.584160 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.585276 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.585728 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.585801 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.585810 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:06.586092 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 13:11:06.586138 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 13:11:06.586478 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.582927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:42865 (local address 127.8.88.129:32915)
0928 13:11:06.583199 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:06.583209 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:06.583261 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:06.583463 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:06.583464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:06.583466 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:06.583587 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.583589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.584259 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.584260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.585177 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:06.585180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.585906 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.585907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:06.585999 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.586004 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:11:06.586071 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 13:11:06.586312 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":17,"thread_start_us":97,"threads_started":1}
I20260928 13:11:06.586445 23136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b165f5a6ab4718b29c231acb6dacbe" candidate_uuid: "1ffea481b6204c6b824aa394070b6f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a662878cdc4402181d95cb0bf679a2b" is_pre_election: true
I20260928 13:11:06.586628 23136 raft_consensus.cc:2454] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ffea481b6204c6b824aa394070b6f79 in term 0.
I20260928 13:11:06.586853 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b165f5a6ab4718b29c231acb6dacbe" candidate_uuid: "1ffea481b6204c6b824aa394070b6f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9963837be23a4215a24c6ac251f5f11a" is_pre_election: true
I20260928 13:11:06.586884 22932 leader_election.cc:302] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ffea481b6204c6b824aa394070b6f79, 7a662878cdc4402181d95cb0bf679a2b; no voters:
I20260928 13:11:06.586971 23268 raft_consensus.cc:2454] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ffea481b6204c6b824aa394070b6f79 in term 0.
I20260928 13:11:06.587023 23050 raft_consensus.cc:2781] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:11:06.587102 23050 raft_consensus.cc:482] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:11:06.587136 23050 raft_consensus.cc:3037] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:06.587992 23050 raft_consensus.cc:504] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:06.588153 23050 leader_election.cc:288] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [CANDIDATE]: Term 1 election: Requested vote from peers 7a662878cdc4402181d95cb0bf679a2b (127.8.88.130:38115), 9963837be23a4215a24c6ac251f5f11a (127.8.88.131:42865)
I20260928 13:11:06.588301 23136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b165f5a6ab4718b29c231acb6dacbe" candidate_uuid: "1ffea481b6204c6b824aa394070b6f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a662878cdc4402181d95cb0bf679a2b"
I20260928 13:11:06.588392 23268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b165f5a6ab4718b29c231acb6dacbe" candidate_uuid: "1ffea481b6204c6b824aa394070b6f79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9963837be23a4215a24c6ac251f5f11a"
I20260928 13:11:06.588482 23268 raft_consensus.cc:3037] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:06.588399 23136 raft_consensus.cc:3037] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:06.589453 23136 raft_consensus.cc:2454] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ffea481b6204c6b824aa394070b6f79 in term 1.
I20260928 13:11:06.589453 23268 raft_consensus.cc:2454] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ffea481b6204c6b824aa394070b6f79 in term 1.
I20260928 13:11:06.589609 22931 leader_election.cc:302] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ffea481b6204c6b824aa394070b6f79, 9963837be23a4215a24c6ac251f5f11a; no voters:
I20260928 13:11:06.589753 23050 raft_consensus.cc:2781] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:11:06.589869 23050 raft_consensus.cc:686] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 1 LEADER]: Becoming Leader. State: Replica: 1ffea481b6204c6b824aa394070b6f79, State: Running, Role: LEADER
I20260928 13:11:06.589967 23050 consensus_queue.cc:237] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:06.590730 22856 catalog_manager.cc:5495] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ffea481b6204c6b824aa394070b6f79 (127.8.88.129). New cstate: current_term: 1 leader_uuid: "1ffea481b6204c6b824aa394070b6f79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } health_report { overall_health: HEALTHY } } }
W20260928 13:11:06.615909 23045 tablet.cc:2369] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:11:06.622376 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.619372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.8.88.129:40083)
0928 13:11:06.619497 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:06.619499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:06.619508 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:11:06.619552 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:06.619553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:06.619575 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:06.619640 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:11:06.619925 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.620472 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.621351 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:06.621562 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:06.621630 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:06.621854 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:11:06.621995 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:11:06.622073 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:06.622208 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 13:11:06.622235 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 13:11:06.624269 23136 raft_consensus.cc:1261] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 1 FOLLOWER]: Refusing update from remote peer 1ffea481b6204c6b824aa394070b6f79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:11:06.624282 23268 raft_consensus.cc:1261] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [term 1 FOLLOWER]: Refusing update from remote peer 1ffea481b6204c6b824aa394070b6f79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:11:06.624682 23050 consensus_queue.cc:1035] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:11:06.624846 23331 consensus_queue.cc:1035] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:11:06.627671 23336 mvcc.cc:205] Tried to move back new op lower bound from 7334301968890241024 to 7334301968753102848. Current Snapshot: MvccSnapshot[applied={T|T < 7334301968890241024}]
I20260928 13:11:06.627713 23335 mvcc.cc:205] Tried to move back new op lower bound from 7334301968890241024 to 7334301968753102848. Current Snapshot: MvccSnapshot[applied={T|T < 7334301968890241024}]
I20260928 13:11:06.628229 23334 mvcc.cc:205] Tried to move back new op lower bound from 7334301968890241024 to 7334301968753102848. Current Snapshot: MvccSnapshot[applied={T|T < 7334301968890241024 or (T in {7334301968890241024})}]
I20260928 13:11:06.629634 22856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:11:06.629735 22856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:11:06.630443 22856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Sending DeleteTablet for 3 replicas of tablet 69b165f5a6ab4718b29c231acb6dacbe
I20260928 13:11:06.630777 22978 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b165f5a6ab4718b29c231acb6dacbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:11:06 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:55402
I20260928 13:11:06.630805 23116 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b165f5a6ab4718b29c231acb6dacbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:11:06 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:45622
I20260928 13:11:06.630872 23053 tablet_replica.cc:330] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: stopping tablet replica
I20260928 13:11:06.630877 23248 tablet_service.cc:1514] Processing DeleteTablet for tablet 69b165f5a6ab4718b29c231acb6dacbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:11:06 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:48246
I20260928 13:11:06.630947 23053 raft_consensus.cc:2229] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:11:06.631029 23053 raft_consensus.cc:2258] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:11:06.631115 23345 tablet_replica.cc:330] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: stopping tablet replica
I20260928 13:11:06.631170 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 22894
I20260928 13:11:06.631209 23345 raft_consensus.cc:2229] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:11:06.631305 23345 raft_consensus.cc:2258] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:11:06.631721 23345 ts_tablet_manager.cc:1905] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:11:06.633566 23345 ts_tablet_manager.cc:1918] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:11:06.633657 23345 log.cc:1196] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/69b165f5a6ab4718b29c231acb6dacbe
I20260928 13:11:06.633947 23345 ts_tablet_manager.cc:1939] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Deleting consensus metadata
I20260928 13:11:06.634366 22841 catalog_manager.cc:4841] TS 9963837be23a4215a24c6ac251f5f11a (127.8.88.131:42865): tablet 69b165f5a6ab4718b29c231acb6dacbe (table test-table [id=1c3b638a3d1b4f8b8927e80b13f141f1]) successfully deleted
W20260928 13:11:06.637044 22842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:40083 (error 108)
I20260928 13:11:06.637373 23344 tablet_replica.cc:330] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: stopping tablet replica
I20260928 13:11:06.637641 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.637438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:40083 (local address 127.0.0.1:55430)
0928 13:11:06.637524 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:06.637605 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:40083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 13:11:06.637641 23344 raft_consensus.cc:2229] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:11:06.637784 23344 raft_consensus.cc:2258] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:11:06.638047 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 23049
W20260928 13:11:06.637883 22842 catalog_manager.cc:4584] TS 1ffea481b6204c6b824aa394070b6f79 (127.8.88.129:40083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69b165f5a6ab4718b29c231acb6dacbe: Network error: Client connection negotiation failed: client connection to 127.8.88.129:40083: connect: Connection refused (error 111)
I20260928 13:11:06.638360 23344 ts_tablet_manager.cc:1905] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:11:06.642818 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 23185
I20260928 13:11:06.642856 23320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:06.642715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:38115 (local address 127.0.0.1:45636)
0928 13:11:06.642774 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:06.642811 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.130:38115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 13:11:06.647393 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 22823
I20260928 13:11:06.652655 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46775
--webserver_interface=127.8.88.190
--webserver_port=35669
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:06.737221 23347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:06.737445 23347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:06.737473 23347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:06.738735 23347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:11:06.738785 23347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:06.738799 23347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:06.738812 23347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:11:06.738824 23347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:11:06.740422 23347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=35669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:06.740686 23347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:06.740985 23347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:06.746243 23352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:06.746250 23353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:06.746250 23355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:06.746356 23347 server_base.cc:1034] running on GCE node
I20260928 13:11:06.746795 23347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:06.747145 23347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:06.748346 23347 hybrid_clock.cc:648] HybridClock initialized: now 1790601066748309 us; error 69 us; skew 500 ppm
Sep 28 13:11:06 dist-test-slave-p1lc krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601066, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:06.751389 23347 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:11:06.751981 23347 webserver.cc:460] Webserver started at http://127.8.88.190:35669/ using document root <none> and password file <none>
I20260928 13:11:06.752247 23347 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:06.752331 23347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:06.753926 23347 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:06.754814 23362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:06.755084 23347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:06.755206 23347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "889d186917ff4470ad633a821f50d51e"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "c73eb95194d98caf9cd67ad66a6c5af2"
server_key_iv: "f041cf8101c58271d83d7f9c039fec2a"
server_key_version: "encryptionkey@0"
I20260928 13:11:06.755640 23347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:06.764755 23347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:06.766997 23347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:06.767189 23347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:06.771883 23347 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:46775
I20260928 13:11:06.771908 23414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:46775 every 8 connection(s)
I20260928 13:11:06.772305 23347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:11:06.774024 23415 sys_catalog.cc:263] Verifying existing consensus state
I20260928 13:11:06.774724 23415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Bootstrap starting.
I20260928 13:11:06.777696 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 23347
I20260928 13:11:06.778009 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:40083
--local_ip_for_outbound_sockets=127.8.88.129
--tserver_master_addrs=127.8.88.190:46775
--webserver_port=39057
--webserver_interface=127.8.88.129
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:11:06.783308 23415 log.cc:824] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:06.785825 23415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:11:06.786150 23415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e: Bootstrap complete.
I20260928 13:11:06.789108 23415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } }
I20260928 13:11:06.789373 23415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889d186917ff4470ad633a821f50d51e, State: Initialized, Role: FOLLOWER
I20260928 13:11:06.789579 23415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } }
I20260928 13:11:06.789676 23415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:11:06.789731 23415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:11:06.789779 23415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 2 FOLLOWER]: Advancing to term 3
I20260928 13:11:06.790777 23415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } }
I20260928 13:11:06.790897 23415 leader_election.cc:302] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 889d186917ff4470ad633a821f50d51e; no voters:
I20260928 13:11:06.791086 23415 leader_election.cc:288] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 13:11:06.791160 23419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 3 FOLLOWER]: Leader election won for term 3
I20260928 13:11:06.791314 23419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [term 3 LEADER]: Becoming Leader. State: Replica: 889d186917ff4470ad633a821f50d51e, State: Running, Role: LEADER
I20260928 13:11:06.791356 23415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:11:06.791489 23419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } }
I20260928 13:11:06.791891 23419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 889d186917ff4470ad633a821f50d51e. Latest consensus state: current_term: 3 leader_uuid: "889d186917ff4470ad633a821f50d51e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } } }
I20260928 13:11:06.791989 23419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:06.792196 23420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "889d186917ff4470ad633a821f50d51e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "889d186917ff4470ad633a821f50d51e" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46775 } } }
I20260928 13:11:06.792279 23420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 889d186917ff4470ad633a821f50d51e [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:06.792503 23428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:11:06.793581 23428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:11:06.793691 23428 catalog_manager.cc:1259] Loaded cluster ID: 78a93f0dfcc94fc4a2d672394ce81067
I20260928 13:11:06.793726 23428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 13:11:06.793967 23428 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f39de5349ad google::LogMessage::Fail() at ??:0
@ 0x7f39de53788a google::LogMessage::SendToLog() at ??:0
@ 0x7f39de5344de google::LogMessage::Flush() at ??:0
@ 0x7f39de535a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f39e64550f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f39e645e8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f39e04020d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f39e03f7954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f39e125b6db start_thread at ??:0
@ 0x7f39dd03261f clone at ??:0
W20260928 13:11:06.903286 23417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:06.903605 23417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:06.903648 23417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:06.905642 23417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:06.905726 23417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:06.905779 23417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:11:06.908257 23417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:40083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=39057
--tserver_master_addrs=127.8.88.190:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:06.908612 23417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:06.908965 23417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:06.915375 23454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:06.915383 23451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:06.915628 23452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:06.915496 23417 server_base.cc:1034] running on GCE node
I20260928 13:11:06.915894 23417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:06.916172 23417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:06.917374 23417 hybrid_clock.cc:648] HybridClock initialized: now 1790601066917353 us; error 38 us; skew 500 ppm
Sep 28 13:11:06 dist-test-slave-p1lc krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601066, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:06.920682 23417 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:11:06.921296 23417 webserver.cc:460] Webserver started at http://127.8.88.129:39057/ using document root <none> and password file <none>
I20260928 13:11:06.921587 23417 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:06.921674 23417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:06.923280 23417 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:11:06.924345 23461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:06.924770 23417 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:11:06.924930 23417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "1ffea481b6204c6b824aa394070b6f79"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "dd85bb84f2a960650cbd4c29224025dd"
server_key_iv: "b9f3953cbd5d12bbd3e3611cc21701f2"
server_key_version: "encryptionkey@0"
I20260928 13:11:06.925390 23417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:06.939394 23417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:06.942243 23417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:06.942492 23417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:06.942790 23417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:06.943287 23468 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 13:11:06.945744 23417 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 13:11:06.945815 23417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260928 13:11:06.945895 23417 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 13:11:06.946769 23417 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 13:11:06.946837 23417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 13:11:06.946926 23468 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: Bootstrap starting.
I20260928 13:11:06.955547 23417 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:40083
I20260928 13:11:06.955802 23575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:40083 every 8 connection(s)
I20260928 13:11:06.956096 23417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:11:06.959568 23468 log.cc:824] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:06.962628 23468 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:11:06.962980 23468 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: Bootstrap complete.
I20260928 13:11:06.964249 23468 ts_tablet_manager.cc:1397] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20260928 13:11:06.964295 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 23417
I20260928 13:11:06.964581 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:38115
--local_ip_for_outbound_sockets=127.8.88.130
--tserver_master_addrs=127.8.88.190:46775
--webserver_port=34267
--webserver_interface=127.8.88.130
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:11:06.965894 23468 raft_consensus.cc:348] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:06.966296 23468 raft_consensus.cc:729] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ffea481b6204c6b824aa394070b6f79, State: Initialized, Role: FOLLOWER
I20260928 13:11:06.966500 23468 consensus_queue.cc:260] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:06.966807 23468 ts_tablet_manager.cc:1428] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: Time spent starting tablet: real 0.002s user 0.003s sys 0.003s
W20260928 13:11:07.047312 23578 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:11:06.957113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:39305)
0928 13:11:06.957312 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:06.957386 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:06.957861 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:07.047085 (+ 89224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":121,"threads_started":1}
W20260928 13:11:07.047737 23576 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:46775 (0 consecutive failures): Network error: Failed to ping master at 127.8.88.190:46775: Client connection negotiation failed: client connection to 127.8.88.190:46775: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 13:11:07.048190 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:07.047986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:53515)
0928 13:11:07.048098 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:07.048162 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 13:11:07.048635 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:07.048539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:44245)
0928 13:11:07.048586 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:07.048613 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 13:11:07.048765 23576 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 13:11:07.052676 23580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:07.052938 23580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:07.052978 23580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:07.054325 23580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:07.054383 23580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:07.054414 23580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:11:07.055946 23580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:38115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=34267
--tserver_master_addrs=127.8.88.190:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:07.056151 23580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:07.056349 23580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:07.061969 23587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:07.061969 23586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:07.061978 23589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:07.062429 23580 server_base.cc:1034] running on GCE node
I20260928 13:11:07.062683 23580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:07.062947 23580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:07.064081 23580 hybrid_clock.cc:648] HybridClock initialized: now 1790601067064066 us; error 30 us; skew 500 ppm
Sep 28 13:11:07 dist-test-slave-p1lc krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601067, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:07.067142 23580 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:11:07.067612 23580 webserver.cc:460] Webserver started at http://127.8.88.130:34267/ using document root <none> and password file <none>
I20260928 13:11:07.067914 23580 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:07.067962 23580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:07.069785 23580 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:11:07.070641 23596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:07.070885 23580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:07.070999 23580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "7a662878cdc4402181d95cb0bf679a2b"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "b899baa509aba22014acb1fa3d454d36"
server_key_iv: "76b9828ece31024832d0f6522c6132ee"
server_key_version: "encryptionkey@0"
I20260928 13:11:07.071411 23580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:07.101176 23580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:07.103571 23580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:07.103762 23580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:07.104028 23580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:07.104542 23603 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 13:11:07.106333 23580 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 13:11:07.106406 23580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 13:11:07.106477 23580 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 13:11:07.107247 23580 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 13:11:07.107318 23580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 13:11:07.107496 23603 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: Bootstrap starting.
I20260928 13:11:07.109493 23603 log.cc:824] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:07.112519 23603 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:11:07.113026 23603 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: Bootstrap complete.
I20260928 13:11:07.113237 23603 ts_tablet_manager.cc:1397] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 13:11:07.114955 23603 raft_consensus.cc:348] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:07.115284 23603 raft_consensus.cc:729] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a662878cdc4402181d95cb0bf679a2b, State: Initialized, Role: FOLLOWER
I20260928 13:11:07.115631 23580 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:38115
I20260928 13:11:07.115720 23711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:38115 every 8 connection(s)
I20260928 13:11:07.115657 23603 consensus_queue.cc:260] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:07.116048 23580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:11:07.116101 23603 ts_tablet_manager.cc:1428] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:11:07.117259 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:07.116777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:51079)
0928 13:11:07.117043 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:07.117119 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":137,"threads_started":1}
W20260928 13:11:07.117458 23713 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:46775 (0 consecutive failures): Network error: Failed to ping master at 127.8.88.190:46775: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
I20260928 13:11:07.117789 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:07.117677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:45647)
0928 13:11:07.117732 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:07.117758 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 13:11:07.118062 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:07.117976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:52819)
0928 13:11:07.118020 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:07.118043 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 13:11:07.118171 23713 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 13:11:07.121137 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 23580
I20260928 13:11:07.121399 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:42865
--local_ip_for_outbound_sockets=127.8.88.131
--tserver_master_addrs=127.8.88.190:46775
--webserver_port=36171
--webserver_interface=127.8.88.131
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:07.214119 23716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:07.214390 23716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:07.214427 23716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:07.215874 23716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:07.215937 23716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:07.215969 23716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:11:07.217852 23716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:42865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=36171
--tserver_master_addrs=127.8.88.190:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:07.218173 23716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:07.218480 23716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:07.224498 23722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:07.224568 23724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:07.224545 23721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:07.224689 23716 server_base.cc:1034] running on GCE node
I20260928 13:11:07.225024 23716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:07.225322 23716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:07.226497 23716 hybrid_clock.cc:648] HybridClock initialized: now 1790601067226472 us; error 46 us; skew 500 ppm
Sep 28 13:11:07 dist-test-slave-p1lc krb5kdc[22276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601067, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:07.229641 23716 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:11:07.230230 23716 webserver.cc:460] Webserver started at http://127.8.88.131:36171/ using document root <none> and password file <none>
I20260928 13:11:07.230552 23716 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:07.230643 23716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:07.232206 23716 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:07.233021 23731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:07.233223 23716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:11:07.233330 23716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "9963837be23a4215a24c6ac251f5f11a"
format_stamp: "Formatted at 2026-09-28 13:11:05 on dist-test-slave-p1lc"
server_key: "e07e143e99eccfb2b1ac43865ea9a46c"
server_key_iv: "daad6d02b782d9f24305a853325a048a"
server_key_version: "encryptionkey@0"
I20260928 13:11:07.233767 23716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:07.243875 23716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:07.246486 23716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:07.246680 23716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:07.246941 23716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:07.247285 23716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:11:07.247344 23716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:07.247386 23716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:11:07.247414 23716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:07.253798 23716 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:42865
I20260928 13:11:07.253841 23844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:42865 every 8 connection(s)
I20260928 13:11:07.254204 23716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:11:07.255280 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:07.254878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:58715)
0928 13:11:07.255058 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:07.255134 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
W20260928 13:11:07.255760 23845 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:46775 (0 consecutive failures): Network error: Failed to ping master at 127.8.88.190:46775: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
I20260928 13:11:07.256102 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:07.256005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:42265)
0928 13:11:07.256057 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:07.256085 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 13:11:07.256654 23847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:07.256520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:38773)
0928 13:11:07.256602 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:07.256631 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 13:11:07.256814 23845 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 13:11:07.257718 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 23716
I20260928 13:11:08.049438 23849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:08.049150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:34719)
0928 13:11:08.049314 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:08.049368 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260928 13:11:08.118857 23850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:08.118596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:57249)
0928 13:11:08.118770 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:08.118815 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":65,"threads_started":1}
I20260928 13:11:08.257524 23851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:08.257294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:51187)
0928 13:11:08.257418 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:08.257463 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260928 13:11:08.686753 23852 raft_consensus.cc:482] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:11:08.686939 23852 raft_consensus.cc:504] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:08.687361 23852 leader_election.cc:288] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a662878cdc4402181d95cb0bf679a2b (127.8.88.130:38115), 9963837be23a4215a24c6ac251f5f11a (127.8.88.131:42865)
Sep 28 13:11:08 dist-test-slave-p1lc krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601066, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.131@KRBTEST.COM
Sep 28 13:11:08 dist-test-slave-p1lc krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601066, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.130@KRBTEST.COM
I20260928 13:11:08.700852 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:08.687592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:42865 (local address 127.8.88.129:50359)
0928 13:11:08.687725 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:08.687739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:08.687806 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:08.688861 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:08.688862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:08.689057 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:08.689199 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:08.689207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:08.690983 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:08.690984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:08.692331 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:08.692335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:08.693256 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:08.693256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:08.693400 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:08.694147 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:08.694158 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:08.696182 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:08.698892 (+ 2710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:08.698900 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:08.698902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:08.699147 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:08.699504 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:08.699505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:08.699505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:08.699656 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:08.700230 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:08.700236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:08.700379 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 13:11:08.700506 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 13:11:08.700732 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":59,"thread_start_us":67,"threads_started":1}
I20260928 13:11:08.701205 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:08.687670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:38115 (local address 127.8.88.129:55441)
0928 13:11:08.687798 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:08.687809 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:08.687867 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:08.688410 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:08.688413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:08.688857 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:08.689200 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:08.689207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:08.690972 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:08.690974 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:08.692331 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:08.692335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:08.693254 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:08.693256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:08.693400 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:08.694147 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:08.694158 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:08.696559 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:08.699225 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:08.699226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:08.699227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:08.699533 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:08.699809 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:08.699809 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:08.699810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:08.699902 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:08.700431 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:08.700432 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:08.700514 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 13:11:08.700604 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 13:11:08.700895 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":192,"thread_start_us":66,"threads_started":1}
I20260928 13:11:08.701412 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:08.687740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:50359 (local address 127.8.88.131:42865)
0928 13:11:08.687915 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:08.687930 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:08.688007 (+ 77us) server_negotiation.cc:407] Connection header received
0928 13:11:08.688651 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:08.688654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:08.688694 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:08.688846 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:08.689348 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:08.690788 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:08.692456 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:08.693133 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:08.693251 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:08.696339 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:08.696345 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:08.696345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:08.696365 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:08.698783 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:08.699276 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:08.699279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:08.699279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:08.699387 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:08.699734 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:08.699735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:08.699735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:08.700130 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:08.701049 (+ 919us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:08.701128 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 13:11:08.701264 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":58,"threads_started":1}
I20260928 13:11:08.701488 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:08.687777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:55441 (local address 127.8.88.130:38115)
0928 13:11:08.687977 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:08.687985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:08.688007 (+ 22us) server_negotiation.cc:407] Connection header received
0928 13:11:08.688138 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:08.688140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:08.688189 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:08.688328 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:08.689343 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:08.690788 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:08.692465 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:08.693133 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:08.693251 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:08.696681 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:08.696686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:08.696686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:08.696699 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:08.699133 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:08.699612 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:08.699615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:08.699616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:08.699735 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:08.699985 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:08.699986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:08.699986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:08.700325 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:08.700410 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:08.700692 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 13:11:08.700827 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":106,"threads_started":1}
I20260928 13:11:08.701862 23666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b165f5a6ab4718b29c231acb6dacbe" candidate_uuid: "1ffea481b6204c6b824aa394070b6f79" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7a662878cdc4402181d95cb0bf679a2b" is_pre_election: true
I20260928 13:11:08.702034 23666 raft_consensus.cc:2454] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ffea481b6204c6b824aa394070b6f79 in term 1.
W20260928 13:11:08.702237 23463 leader_election.cc:341] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9963837be23a4215a24c6ac251f5f11a (127.8.88.131:42865): Not found: Tablet not found: 69b165f5a6ab4718b29c231acb6dacbe
I20260928 13:11:08.702359 23464 leader_election.cc:302] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ffea481b6204c6b824aa394070b6f79, 7a662878cdc4402181d95cb0bf679a2b; no voters: 9963837be23a4215a24c6ac251f5f11a
I20260928 13:11:08.702498 23852 raft_consensus.cc:2781] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 13:11:08.702553 23852 raft_consensus.cc:482] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:11:08.702569 23852 raft_consensus.cc:3037] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 1 FOLLOWER]: Advancing to term 2
I20260928 13:11:08.703802 23852 raft_consensus.cc:504] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:08.703990 23852 leader_election.cc:288] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [CANDIDATE]: Term 2 election: Requested vote from peers 7a662878cdc4402181d95cb0bf679a2b (127.8.88.130:38115), 9963837be23a4215a24c6ac251f5f11a (127.8.88.131:42865)
I20260928 13:11:08.704259 23666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69b165f5a6ab4718b29c231acb6dacbe" candidate_uuid: "1ffea481b6204c6b824aa394070b6f79" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7a662878cdc4402181d95cb0bf679a2b"
I20260928 13:11:08.704362 23666 raft_consensus.cc:3037] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 1 FOLLOWER]: Advancing to term 2
W20260928 13:11:08.704334 23463 leader_election.cc:341] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 9963837be23a4215a24c6ac251f5f11a (127.8.88.131:42865): Not found: Tablet not found: 69b165f5a6ab4718b29c231acb6dacbe
I20260928 13:11:08.705780 23666 raft_consensus.cc:2454] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ffea481b6204c6b824aa394070b6f79 in term 2.
I20260928 13:11:08.706107 23464 leader_election.cc:302] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ffea481b6204c6b824aa394070b6f79, 7a662878cdc4402181d95cb0bf679a2b; no voters: 9963837be23a4215a24c6ac251f5f11a
I20260928 13:11:08.706251 23852 raft_consensus.cc:2781] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 13:11:08.706394 23852 raft_consensus.cc:686] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [term 2 LEADER]: Becoming Leader. State: Replica: 1ffea481b6204c6b824aa394070b6f79, State: Running, Role: LEADER
I20260928 13:11:08.706519 23852 consensus_queue.cc:237] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:08.706871 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:08.706744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:36035)
0928 13:11:08.706790 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:08.706835 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 13:11:09.119958 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:09.119666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:58847)
0928 13:11:09.119850 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:09.119891 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
W20260928 13:11:09.250499 23463 consensus_peers.cc:487] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 -> Peer 9963837be23a4215a24c6ac251f5f11a (127.8.88.131:42865): Couldn't send request to peer 9963837be23a4215a24c6ac251f5f11a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69b165f5a6ab4718b29c231acb6dacbe. This is attempt 1: this message will repeat every 5th retry.
I20260928 13:11:09.258356 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:09.258121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:49461)
0928 13:11:09.258269 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:09.258307 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260928 13:11:09.284394 23666 raft_consensus.cc:1261] T 69b165f5a6ab4718b29c231acb6dacbe P 7a662878cdc4402181d95cb0bf679a2b [term 2 FOLLOWER]: Refusing update from remote peer 1ffea481b6204c6b824aa394070b6f79: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 13:11:09.284920 23860 consensus_queue.cc:1035] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 13:11:09.286095 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:09.285928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:35161)
0928 13:11:09.286015 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:09.286060 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 13:11:09.287101 23867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:09.286907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:39541)
0928 13:11:09.287022 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:09.287054 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260928 13:11:09.287202 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:09.287062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:39753)
0928 13:11:09.287114 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:09.287166 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 13:11:09.692034 23868 ts_tablet_manager.cc:927] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Initiating tablet copy from peer 1ffea481b6204c6b824aa394070b6f79 (127.8.88.129:40083)
I20260928 13:11:09.692317 23868 tablet_copy_client.cc:316] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: tablet copy: Beginning tablet copy session from remote peer at address 127.8.88.129:40083
Sep 28 13:11:09 dist-test-slave-p1lc krb5kdc[22276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601067, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.129@KRBTEST.COM
I20260928 13:11:09.701645 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:09.692769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:41105 (local address 127.8.88.129:40083)
0928 13:11:09.692986 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:09.692990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:09.693001 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:11:09.693034 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:09.693036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:09.693049 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:09.693121 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:09.693688 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:09.694421 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:09.695243 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:09.695591 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:09.695655 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:09.698177 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:09.698187 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:09.698187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:09.698195 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:09.700187 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:09.700651 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:09.700655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:09.700655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:09.700724 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:09.701049 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:09.701050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:09.701050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:09.701217 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:09.701287 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:09.701488 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 13:11:09.701562 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":120,"threads_started":1}
I20260928 13:11:09.701730 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:09.692658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:40083 (local address 127.8.88.131:41105)
0928 13:11:09.692739 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:09.692761 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:09.692843 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:09.693191 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:09.693198 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:09.693541 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:09.693602 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:09.693605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:09.694533 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:09.694534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:09.695145 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:09.695147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:09.695682 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:09.695684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:09.695757 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:09.696465 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:09.696471 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:09.698055 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:09.700291 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:09.700299 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:09.700301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:09.700540 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:09.700814 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:09.700815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:09.700816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:09.700963 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:09.701305 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:09.701312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:09.701366 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 13:11:09.701528 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 13:11:09.701646 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 13:11:09.702399 23550 tablet_copy_service.cc:140] P 1ffea481b6204c6b824aa394070b6f79: Received BeginTabletCopySession request for tablet 69b165f5a6ab4718b29c231acb6dacbe from peer 9963837be23a4215a24c6ac251f5f11a ({username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:41105)
I20260928 13:11:09.702477 23550 tablet_copy_service.cc:161] P 1ffea481b6204c6b824aa394070b6f79: Beginning new tablet copy session on tablet 69b165f5a6ab4718b29c231acb6dacbe from peer 9963837be23a4215a24c6ac251f5f11a at {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:41105: session id = 9963837be23a4215a24c6ac251f5f11a-69b165f5a6ab4718b29c231acb6dacbe
I20260928 13:11:09.703015 23550 tablet_copy_source_session.cc:205] T 69b165f5a6ab4718b29c231acb6dacbe P 1ffea481b6204c6b824aa394070b6f79: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 13:11:09.704108 23868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69b165f5a6ab4718b29c231acb6dacbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:09.707619 23868 tablet_copy_client.cc:791] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: tablet copy: Starting download of 0 data blocks...
I20260928 13:11:09.707891 23868 tablet_copy_client.cc:655] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: tablet copy: Starting download of 1 WAL segments...
I20260928 13:11:09.709126 23868 tablet_copy_client.cc:531] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 13:11:09.710002 23868 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Bootstrap starting.
I20260928 13:11:09.711462 23868 log.cc:824] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:09.714012 23868 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:11:09.714219 23868 tablet_bootstrap.cc:492] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Bootstrap complete.
I20260928 13:11:09.714322 23868 ts_tablet_manager.cc:1397] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260928 13:11:09.715917 23868 raft_consensus.cc:348] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:09.716078 23868 raft_consensus.cc:729] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9963837be23a4215a24c6ac251f5f11a, State: Initialized, Role: FOLLOWER
I20260928 13:11:09.716219 23868 consensus_queue.cc:260] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a662878cdc4402181d95cb0bf679a2b" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38115 } } peers { permanent_uuid: "9963837be23a4215a24c6ac251f5f11a" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 42865 } } peers { permanent_uuid: "1ffea481b6204c6b824aa394070b6f79" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 40083 } }
I20260928 13:11:09.716535 23868 ts_tablet_manager.cc:1428] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260928 13:11:09.716976 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:09.716781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:54137)
0928 13:11:09.716855 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:09.716935 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 13:11:09.717044 23550 tablet_copy_service.cc:342] P 1ffea481b6204c6b824aa394070b6f79: Request end of tablet copy session 9963837be23a4215a24c6ac251f5f11a-69b165f5a6ab4718b29c231acb6dacbe received from {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:41105
I20260928 13:11:09.717123 23550 tablet_copy_service.cc:433] P 1ffea481b6204c6b824aa394070b6f79: ending tablet copy session 9963837be23a4215a24c6ac251f5f11a-69b165f5a6ab4718b29c231acb6dacbe on tablet 69b165f5a6ab4718b29c231acb6dacbe with peer 9963837be23a4215a24c6ac251f5f11a
I20260928 13:11:09.717358 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:09.717242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:39389)
0928 13:11:09.717292 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:09.717322 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 13:11:10.235308 23799 raft_consensus.cc:1203] T 69b165f5a6ab4718b29c231acb6dacbe P 9963837be23a4215a24c6ac251f5f11a [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 13:11:10.235432 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:10.235162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:44249)
0928 13:11:10.235269 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:10.235311 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":67,"threads_started":1}
I20260928 13:11:10.236034 23876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:10.235820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:47003)
0928 13:11:10.235937 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:10.235980 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260928 13:11:10.287883 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:10.287668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:52109)
0928 13:11:10.287797 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:10.287835 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260928 13:11:11.236187 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:11.235950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:60809)
0928 13:11:11.236099 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:11.236140 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260928 13:11:11.236799 23879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:11.236583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:39883)
0928 13:11:11.236727 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:11.236763 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260928 13:11:11.288656 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:11.288418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:41835)
0928 13:11:11.288570 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:11.288608 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260928 13:11:12.237133 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:12.236853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:50979)
0928 13:11:12.237047 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:12.237084 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":117,"threads_started":1}
I20260928 13:11:12.237443 23882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:12.237244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:33573)
0928 13:11:12.237353 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:12.237391 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260928 13:11:12.237603 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:12.237426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:37857)
0928 13:11:12.237538 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:12.237576 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 13:11:12.289672 23883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:12.289375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:47409)
0928 13:11:12.289561 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:12.289609 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
I20260928 13:11:13.238286 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:13.238081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:49979)
0928 13:11:13.238200 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:13.238239 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":51,"threads_started":1}
I20260928 13:11:13.238283 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:13.238034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:45607)
0928 13:11:13.238190 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:13.238232 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260928 13:11:13.290473 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:13.290228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:39921)
0928 13:11:13.290374 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:13.290424 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260928 13:11:14.239135 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:14.238881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:47657)
0928 13:11:14.239046 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:14.239087 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":63,"threads_started":1}
I20260928 13:11:14.239151 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:14.238888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:53675)
0928 13:11:14.239056 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:14.239108 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":109,"threads_started":1}
I20260928 13:11:14.291301 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:14.291053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:57845)
0928 13:11:14.291205 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:14.291252 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260928 13:11:15.240061 23890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:15.239805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:33081)
0928 13:11:15.239964 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:15.240014 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260928 13:11:15.240067 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:15.239805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:39135)
0928 13:11:15.239983 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:15.240016 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260928 13:11:15.292263 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:15.291997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:45797)
0928 13:11:15.292171 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:15.292215 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":97,"threads_started":1}
I20260928 13:11:16.241025 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:16.240712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:40441)
0928 13:11:16.240922 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:16.240977 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260928 13:11:16.241055 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:16.240710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:53897)
0928 13:11:16.240927 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:16.240976 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":92,"threads_started":1}
I20260928 13:11:16.293233 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:16.292848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:46813)
0928 13:11:16.293079 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:16.293135 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":161,"threads_started":1}
I20260928 13:11:17.241822 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:17.241579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:46833)
0928 13:11:17.241718 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:17.241765 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260928 13:11:17.241819 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:17.241579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:39157)
0928 13:11:17.241746 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:17.241775 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
I20260928 13:11:17.294159 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:17.293912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:52511)
0928 13:11:17.294068 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:17.294109 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260928 13:11:18.242740 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:18.242458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:37327)
0928 13:11:18.242651 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:18.242693 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260928 13:11:18.242754 23901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:18.242458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:58125)
0928 13:11:18.242616 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:18.242713 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260928 13:11:18.295046 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:18.294753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:41799)
0928 13:11:18.294937 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:18.294997 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":74,"threads_started":1}
I20260928 13:11:19.243633 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:19.243364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:44171)
0928 13:11:19.243519 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:19.243568 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260928 13:11:19.243637 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:19.243364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:44789)
0928 13:11:19.243548 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:19.243585 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20260928 13:11:19.296103 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:19.295831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:57063)
0928 13:11:19.296005 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:19.296052 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260928 13:11:20.244632 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:20.244302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:41771)
0928 13:11:20.244492 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:20.244538 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":83,"threads_started":1}
I20260928 13:11:20.244637 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:20.244302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:47309)
0928 13:11:20.244514 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:20.244570 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
I20260928 13:11:20.297135 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:20.296712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:37799)
0928 13:11:20.297033 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:20.297087 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":85,"threads_started":1}
I20260928 13:11:21.245517 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:21.245276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:33303)
0928 13:11:21.245417 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:21.245469 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260928 13:11:21.245527 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:21.245296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:60843)
0928 13:11:21.245450 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:21.245487 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260928 13:11:21.297981 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:21.297684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.130:37557)
0928 13:11:21.297856 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:21.297902 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20260928 13:11:22.246342 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:22.246121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.131:45655)
0928 13:11:22.246264 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:22.246294 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260928 13:11:22.246348 23913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:22.246122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46775 (local address 127.8.88.129:41505)
0928 13:11:22.246266 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:22.246306 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260928 13:11:22.263957 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 23417
I20260928 13:11:22.269505 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 23580
I20260928 13:11:22.273828 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 23716
2026-09-28T13:11:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18520 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:11:22 dist-test-slave-p1lc krb5kdc[23916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:11:22 dist-test-slave-p1lc krb5kdc[23916](info): set up 2 sockets
Sep 28 13:11:22 dist-test-slave-p1lc krb5kdc[23916](info): commencing operation
krb5kdc: starting...
W20260928 13:11:23.329319 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:11:23 dist-test-slave-p1lc krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:11:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:11:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:11:23.452685 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:39279
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:39279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:23.535308 23929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:23.535530 23929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:23.535553 23929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:23.536787 23929 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 13:11:23.536831 23929 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 13:11:23.536845 23929 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 13:11:23.536857 23929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:11:23.536927 23929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:23.536955 23929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:23.536968 23929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:11:23.536981 23929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:11:23.538547 23929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:39279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:39279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:23.538762 23929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:23.538934 23929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:23.544515 23934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:23.544626 23929 server_base.cc:1034] running on GCE node
W20260928 13:11:23.544670 23935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:23.544670 23937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:23.545212 23929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:23.545483 23929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:23.546643 23929 hybrid_clock.cc:648] HybridClock initialized: now 1790601083546619 us; error 43 us; skew 500 ppm
Sep 28 13:11:23 dist-test-slave-p1lc krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601083, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:23.557291 23929 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:11:23.557801 23929 webserver.cc:460] Webserver started at http://127.8.88.190:40423/ using document root <none> and password file <none>
I20260928 13:11:23.558048 23929 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:23.558121 23929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:23.558271 23929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:11:23.559386 23929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "13036be9c653434c8519b9fb6d728ed7"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "83189544689d3e20645f43c8313f33c6"
server_key_iv: "4ee62a8ae6c368fb076ce82b8c7db915"
server_key_version: "encryptionkey@0"
I20260928 13:11:23.559821 23929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "13036be9c653434c8519b9fb6d728ed7"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "83189544689d3e20645f43c8313f33c6"
server_key_iv: "4ee62a8ae6c368fb076ce82b8c7db915"
server_key_version: "encryptionkey@0"
I20260928 13:11:23.561380 23929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:11:23.562248 23944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:23.562460 23929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:11:23.562572 23929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "13036be9c653434c8519b9fb6d728ed7"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "83189544689d3e20645f43c8313f33c6"
server_key_iv: "4ee62a8ae6c368fb076ce82b8c7db915"
server_key_version: "encryptionkey@0"
I20260928 13:11:23.562677 23929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:23.576805 23929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:23.579283 23929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:23.579457 23929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:23.583722 23929 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:39279
I20260928 13:11:23.583758 23996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:39279 every 8 connection(s)
I20260928 13:11:23.584139 23929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:11:23.585126 23997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:23.587507 23997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Bootstrap starting.
I20260928 13:11:23.587997 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 23929
I20260928 13:11:23.588090 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:11:23.588284 8546 external_mini_cluster.cc:1428] Setting key a932bf6e42b7140a4e7569e21b1519ec
I20260928 13:11:23.588315 23997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:23.588639 23997 log.cc:824] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:23.589504 23997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: No bootstrap required, opened a new log
I20260928 13:11:23.591917 23997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } }
I20260928 13:11:23.592130 23997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:23.592183 23997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13036be9c653434c8519b9fb6d728ed7, State: Initialized, Role: FOLLOWER
I20260928 13:11:23.592358 23997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } }
I20260928 13:11:23.592453 23997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:11:23.592497 23997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:11:23.592546 23997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:23.593206 23997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } }
I20260928 13:11:23.593338 23997 leader_election.cc:302] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13036be9c653434c8519b9fb6d728ed7; no voters:
I20260928 13:11:23.593536 23997 leader_election.cc:288] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:11:23.593564 24002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:11:23.593745 24002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 1 LEADER]: Becoming Leader. State: Replica: 13036be9c653434c8519b9fb6d728ed7, State: Running, Role: LEADER
I20260928 13:11:23.593873 23997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:11:23.593894 24002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } }
I20260928 13:11:23.594336 24004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13036be9c653434c8519b9fb6d728ed7. Latest consensus state: current_term: 1 leader_uuid: "13036be9c653434c8519b9fb6d728ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } } }
I20260928 13:11:23.594429 24004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:23.594666 24003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13036be9c653434c8519b9fb6d728ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } } }
I20260928 13:11:23.594789 24003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:23.595192 24010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 13:11:23 dist-test-slave-p1lc krb5kdc[23916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:23.595804 24010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:11:23.597705 24010 catalog_manager.cc:1347] Generated new cluster ID: 86a882fa714446208acf1ac8eee87a78
I20260928 13:11:23.597759 24010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:11:23.600227 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:23.588798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.8.88.190:39279)
0928 13:11:23.588952 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:23.588962 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:23.588997 (+ 35us) server_negotiation.cc:407] Connection header received
0928 13:11:23.589559 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:23.589569 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:23.589643 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:23.589811 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:23.590440 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:23.591913 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:23.592962 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:23.593403 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:23.593520 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:23.596421 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:23.596425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:23.596426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:23.596438 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:23.598527 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:23.598924 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:23.598928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:23.598928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:23.599040 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:23.599253 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:23.599254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:23.599255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:23.599584 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:23.599741 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:23.599872 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 13:11:23.600014 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260928 13:11:23.609261 24010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:11:23.609798 24010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:11:23.613847 24010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Generated new TSK 0
I20260928 13:11:23.614039 24010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:11:23.643770 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:39279
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:23.730377 24023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:23.730600 24023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:23.730625 24023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:23.731805 24023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:23.731850 24023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:23.731877 24023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:11:23.733342 24023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:39279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:23.733553 24023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:23.733744 24023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:23.739048 24028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:23.739034 24031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:23.739133 24029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:23.739259 24023 server_base.cc:1034] running on GCE node
I20260928 13:11:23.739598 24023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:23.739871 24023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:23.741061 24023 hybrid_clock.cc:648] HybridClock initialized: now 1790601083741031 us; error 35 us; skew 500 ppm
Sep 28 13:11:23 dist-test-slave-p1lc krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601083, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:23.743800 24023 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:11:23.744262 24023 webserver.cc:460] Webserver started at http://127.8.88.129:46677/ using document root <none> and password file <none>
I20260928 13:11:23.744508 24023 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:23.744578 24023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:23.744720 24023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:11:23.745872 24023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "e5d3d782e890437cb4b1b3d9bc79dccd"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "ba1c485a311d596e57f3c6c2cfa67ab2"
server_key_iv: "be819a1282fc24f32e7c05f7f6d1a1ab"
server_key_version: "encryptionkey@0"
I20260928 13:11:23.746246 24023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "e5d3d782e890437cb4b1b3d9bc79dccd"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "ba1c485a311d596e57f3c6c2cfa67ab2"
server_key_iv: "be819a1282fc24f32e7c05f7f6d1a1ab"
server_key_version: "encryptionkey@0"
I20260928 13:11:23.747646 24023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:11:23.748440 24038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:23.748701 24023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:23.748800 24023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "e5d3d782e890437cb4b1b3d9bc79dccd"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "ba1c485a311d596e57f3c6c2cfa67ab2"
server_key_iv: "be819a1282fc24f32e7c05f7f6d1a1ab"
server_key_version: "encryptionkey@0"
I20260928 13:11:23.748857 24023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:23.757840 24023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:23.760386 24023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:23.760509 24023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:23.760694 24023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:23.761062 24023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:11:23.761099 24023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:23.761123 24023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:11:23.761168 24023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:23.767374 24023 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:46629
I20260928 13:11:23.767452 24151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:46629 every 8 connection(s)
I20260928 13:11:23.767781 24023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:11:23.769158 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 24023
I20260928 13:11:23.769277 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:11:23.769471 8546 external_mini_cluster.cc:1428] Setting key 903662701b3773447dd9ece8e58c5098
Sep 28 13:11:23 dist-test-slave-p1lc krb5kdc[23916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601083, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
I20260928 13:11:23.779215 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:23.768595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:60585 (local address 127.8.88.190:39279)
0928 13:11:23.768642 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:23.768644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:23.768972 (+ 328us) server_negotiation.cc:407] Connection header received
0928 13:11:23.769750 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:23.769752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:23.769767 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:23.769835 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:23.771220 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:23.771840 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:23.772937 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:23.773212 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:23.773276 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:23.776026 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:23.776028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:23.776028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:23.776037 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:23.777695 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:23.778184 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:23.778185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:23.778185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:23.778248 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:23.778515 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:23.778516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:23.778517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:23.778649 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:23.778722 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:23.779016 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 13:11:23.779112 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:11:23.779542 24154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:23.768635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:60585)
0928 13:11:23.768796 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:23.768835 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:23.769621 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:23.769918 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:23.769923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:23.770755 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:23.771082 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:23.771090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:23.771950 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:23.771951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:23.772785 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:23.772788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:23.773306 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:23.773308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:23.773474 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:23.774068 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:23.774079 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:23.775903 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:23.777810 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:23.777820 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:23.777822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:23.778073 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:23.778321 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:23.778322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:23.778322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:23.778443 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:23.778726 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:23.778734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:23.778881 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 13:11:23.778996 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 13:11:23.779182 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260928 13:11:23.780138 24152 heartbeater.cc:347] Connected to a master server at 127.8.88.190:39279
I20260928 13:11:23.780231 24152 heartbeater.cc:464] Registering TS with master...
I20260928 13:11:23.780448 24152 heartbeater.cc:511] Master 127.8.88.190:39279 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
I20260928 13:11:23.781196 23961 ts_manager.cc:194] Registered new tserver with Master: e5d3d782e890437cb4b1b3d9bc79dccd (127.8.88.129:46629)
I20260928 13:11:23.782161 23961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:60585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:11:23.799779 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:39279
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:23.889189 24157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:23.889411 24157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:23.889436 24157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:23.890708 24157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:23.890761 24157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:23.890790 24157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:11:23.892347 24157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:39279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:23.892621 24157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:23.892905 24157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:23.898422 24162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:23.898548 24157 server_base.cc:1034] running on GCE node
W20260928 13:11:23.898428 24163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:23.898418 24165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:23.898898 24157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:23.899178 24157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:23.900377 24157 hybrid_clock.cc:648] HybridClock initialized: now 1790601083900369 us; error 37 us; skew 500 ppm
Sep 28 13:11:23 dist-test-slave-p1lc krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601083, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:23.903250 24157 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:11:23.903777 24157 webserver.cc:460] Webserver started at http://127.8.88.130:42177/ using document root <none> and password file <none>
I20260928 13:11:23.904038 24157 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:23.904116 24157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:23.904271 24157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:11:23.905431 24157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "993e0001b04a455d8a728975e202216d"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "c0a62b67c339270c3479ac705484ed7d"
server_key_iv: "3c3e21116a736bad5a9cd03b319fb46b"
server_key_version: "encryptionkey@0"
I20260928 13:11:23.905798 24157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "993e0001b04a455d8a728975e202216d"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "c0a62b67c339270c3479ac705484ed7d"
server_key_iv: "3c3e21116a736bad5a9cd03b319fb46b"
server_key_version: "encryptionkey@0"
I20260928 13:11:23.907158 24157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:11:23.907886 24172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:23.908074 24157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:11:23.908188 24157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "993e0001b04a455d8a728975e202216d"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "c0a62b67c339270c3479ac705484ed7d"
server_key_iv: "3c3e21116a736bad5a9cd03b319fb46b"
server_key_version: "encryptionkey@0"
I20260928 13:11:23.908272 24157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:23.945837 24157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:23.947932 24157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:23.948113 24157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:23.948374 24157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:23.948714 24157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:11:23.948769 24157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:23.948812 24157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:11:23.948853 24157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:23.956446 24157 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:42593
I20260928 13:11:23.956487 24285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:42593 every 8 connection(s)
I20260928 13:11:23.956830 24157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
Sep 28 13:11:23 dist-test-slave-p1lc krb5kdc[23916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601083, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:23.965766 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 24157
I20260928 13:11:23.965903 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:11:23.966132 8546 external_mini_cluster.cc:1428] Setting key ea8c014de9130d261e53865a7eaec757
I20260928 13:11:23.966581 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:23.957658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:60065 (local address 127.8.88.190:39279)
0928 13:11:23.957718 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:23.957720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:23.957899 (+ 179us) server_negotiation.cc:407] Connection header received
0928 13:11:23.958372 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:23.958372 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:23.958382 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:23.958442 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:23.959388 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:23.959908 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:23.960977 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:23.961187 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:23.961282 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:23.963484 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:23.963486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:23.963486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:23.963492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:23.965070 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:23.965467 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:23.965468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:23.965468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:23.965522 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:23.965852 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:23.965853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:23.965853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:23.966006 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:23.966083 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:23.966406 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 13:11:23.966491 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 13:11:23.966823 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:23.957616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:60065)
0928 13:11:23.957772 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:23.957814 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:23.958291 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:23.958497 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:23.958500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:23.959070 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:23.959272 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:23.959277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:23.960000 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:23.960001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:23.960848 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:23.960850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:23.961338 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:23.961339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:23.961461 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:23.961988 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:23.961997 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:23.963400 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:23.965160 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:23.965166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:23.965168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:23.965390 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:23.965664 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:23.965664 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:23.965665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:23.965779 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:23.966084 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:23.966089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:23.966242 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 13:11:23.966379 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 13:11:23.966546 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260928 13:11:23.967247 24286 heartbeater.cc:347] Connected to a master server at 127.8.88.190:39279
I20260928 13:11:23.967355 24286 heartbeater.cc:464] Registering TS with master...
I20260928 13:11:23.967559 24286 heartbeater.cc:511] Master 127.8.88.190:39279 requested a full tablet report, sending...
I20260928 13:11:23.967965 23961 ts_manager.cc:194] Registered new tserver with Master: 993e0001b04a455d8a728975e202216d (127.8.88.130:42593)
I20260928 13:11:23.968469 23961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:60065
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:11:23.992843 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:39279
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:24.076757 24291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:24.077018 24291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:24.077044 24291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:24.078296 24291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:24.078346 24291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:24.078374 24291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:11:24.079922 24291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:39279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:24.080132 24291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:24.080318 24291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:24.085517 24296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:24.085704 24291 server_base.cc:1034] running on GCE node
W20260928 13:11:24.085517 24299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:24.085546 24297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:24.086007 24291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:24.086256 24291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:24.087399 24291 hybrid_clock.cc:648] HybridClock initialized: now 1790601084087381 us; error 33 us; skew 500 ppm
Sep 28 13:11:24 dist-test-slave-p1lc krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601084, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:24.090164 24291 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:11:24.090672 24291 webserver.cc:460] Webserver started at http://127.8.88.131:44687/ using document root <none> and password file <none>
I20260928 13:11:24.090905 24291 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:24.090950 24291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:24.091114 24291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:11:24.092217 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "4d3546ec45344433ab0fc892c015be7f"
format_stamp: "Formatted at 2026-09-28 13:11:24 on dist-test-slave-p1lc"
server_key: "bde0d300255cd76be335c62ab06d69e0"
server_key_iv: "c0a71a52b3f06e7dba3c62d0f70571f6"
server_key_version: "encryptionkey@0"
I20260928 13:11:24.092590 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "4d3546ec45344433ab0fc892c015be7f"
format_stamp: "Formatted at 2026-09-28 13:11:24 on dist-test-slave-p1lc"
server_key: "bde0d300255cd76be335c62ab06d69e0"
server_key_iv: "c0a71a52b3f06e7dba3c62d0f70571f6"
server_key_version: "encryptionkey@0"
I20260928 13:11:24.094123 24291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 13:11:24.094997 24306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:24.095212 24291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:11:24.095314 24291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "4d3546ec45344433ab0fc892c015be7f"
format_stamp: "Formatted at 2026-09-28 13:11:24 on dist-test-slave-p1lc"
server_key: "bde0d300255cd76be335c62ab06d69e0"
server_key_iv: "c0a71a52b3f06e7dba3c62d0f70571f6"
server_key_version: "encryptionkey@0"
I20260928 13:11:24.095405 24291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:24.104400 24291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:24.106401 24291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:24.106565 24291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:24.106813 24291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:24.107113 24291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:11:24.107167 24291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:24.107219 24291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:11:24.107251 24291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:24.113149 24291 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:37187
I20260928 13:11:24.113209 24419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:37187 every 8 connection(s)
I20260928 13:11:24.113567 24291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:11:24.118589 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 24291
I20260928 13:11:24.118722 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:11:24.118971 8546 external_mini_cluster.cc:1428] Setting key 97caf92a0f76fd41c91fec009a4743ca
Sep 28 13:11:24 dist-test-slave-p1lc krb5kdc[23916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601084, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:24.123611 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.114343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:53707 (local address 127.8.88.190:39279)
0928 13:11:24.114415 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:24.114417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:24.114642 (+ 225us) server_negotiation.cc:407] Connection header received
0928 13:11:24.115142 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:24.115143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:24.115154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:24.115214 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:24.116213 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.116803 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.117901 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.118137 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.118234 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.120581 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:24.120582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:24.120583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:24.120591 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:24.122255 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:24.122635 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:24.122636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:24.122636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:24.122724 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:24.123033 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:24.123034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:24.123034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:24.123148 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:24.123209 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:24.123474 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 13:11:24.123542 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 13:11:24.123926 24422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.114333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:53707)
0928 13:11:24.114514 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:24.114551 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:24.115046 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:24.115270 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:24.115273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:24.115921 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:24.116114 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:24.116119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.116930 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.116931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:24.117761 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:24.117763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.118263 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.118264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:24.118366 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.118942 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:24.118950 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:24.120436 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:24.122335 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:24.122342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:24.122344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:24.122565 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:24.122834 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:24.122835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:24.122835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:24.122952 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:24.123219 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:24.123225 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:24.123353 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 13:11:24.123445 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 13:11:24.123656 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
I20260928 13:11:24.124439 24420 heartbeater.cc:347] Connected to a master server at 127.8.88.190:39279
I20260928 13:11:24.124552 24420 heartbeater.cc:464] Registering TS with master...
I20260928 13:11:24.124752 24420 heartbeater.cc:511] Master 127.8.88.190:39279 requested a full tablet report, sending...
I20260928 13:11:24.125259 23961 ts_manager.cc:194] Registered new tserver with Master: 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187)
I20260928 13:11:24.125639 23961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:53707
I20260928 13:11:24.130894 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:11:24.138862 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.132295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.8.88.190:39279)
0928 13:11:24.132380 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:24.132383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:24.132478 (+ 95us) server_negotiation.cc:407] Connection header received
0928 13:11:24.132569 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:24.132570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:24.132581 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:24.132664 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:24.133303 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.133809 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.134549 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.134746 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.134850 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.136110 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:24.136112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:24.136112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:24.136119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:24.137666 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:24.138034 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:24.138035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:24.138035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:24.138092 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:24.138290 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:24.138291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:24.138291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:24.138448 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:24.138557 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:24.138668 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 13:11:24.138757 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 13:11:24.140502 23961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:11:24.142187 23961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:11:24.148722 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.144790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.8.88.131:37187)
0928 13:11:24.145078 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:24.145082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:24.145091 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:11:24.145128 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:24.145130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:24.145144 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:24.145214 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:24.145475 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.146568 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.147807 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.148333 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.148410 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.148469 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:24.148589 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 13:11:24.148629 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
I20260928 13:11:24.148718 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.144569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:42593 (local address 127.0.0.1:36512)
0928 13:11:24.145004 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:24.145014 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:24.145054 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:24.145302 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:24.145303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:24.145306 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:24.145394 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:24.145398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.146294 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.146295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:24.147655 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:24.147657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.148347 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.148348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:24.148431 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.148444 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 13:11:24.148524 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 13:11:24.148624 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":8,"thread_start_us":103,"threads_started":1}
I20260928 13:11:24.148737 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.144566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:37187 (local address 127.0.0.1:47936)
0928 13:11:24.144961 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:24.144970 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:24.145018 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:24.145300 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:24.145300 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:24.145304 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:24.145394 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:24.145396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.146688 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.146688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:24.147718 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:24.147720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.148429 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.148431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:24.148500 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.148504 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:11:24.148559 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 13:11:24.148628 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":37,"thread_start_us":46,"threads_started":1}
I20260928 13:11:24.149061 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.144687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.8.88.130:42593)
0928 13:11:24.144918 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:24.144922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:24.145092 (+ 170us) server_negotiation.cc:407] Connection header received
0928 13:11:24.145134 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:24.145137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:24.145153 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:24.145223 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:24.145473 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.146196 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.147752 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.148261 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.148318 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.148359 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:24.148941 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 13:11:24.148987 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260928 13:11:24.149784 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.144566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:46629 (local address 127.0.0.1:34626)
0928 13:11:24.144747 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:24.144789 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:24.144858 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:24.145296 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:24.145300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:24.145302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:24.145388 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:24.145389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.146723 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.146724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:24.147658 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:24.147660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.149562 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.149563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:24.149636 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.149640 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:11:24.149671 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 13:11:24.149713 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":2,"thread_start_us":101,"threads_started":1}
I20260928 13:11:24.149775 24220 tablet_service.cc:1467] Processing CreateTablet for tablet 59a2ecbd0fbf4bdb81b1548a151648d0 (DEFAULT_TABLE table=test-table [id=14011b244ac04780babaae7c282ac468]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:11:24.149863 24354 tablet_service.cc:1467] Processing CreateTablet for tablet 59a2ecbd0fbf4bdb81b1548a151648d0 (DEFAULT_TABLE table=test-table [id=14011b244ac04780babaae7c282ac468]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:11:24.150125 24436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.144853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.8.88.129:46629)
0928 13:11:24.145019 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:24.145024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:24.145037 (+ 13us) server_negotiation.cc:407] Connection header received
0928 13:11:24.145080 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:24.145082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:24.145100 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:24.145186 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:24.145476 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.146600 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.148426 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.149413 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.149892 (+ 479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.149963 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:24.149991 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 13:11:24.150033 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260928 13:11:24.150151 24220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a2ecbd0fbf4bdb81b1548a151648d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:24.150168 24354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a2ecbd0fbf4bdb81b1548a151648d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:24.150772 24086 tablet_service.cc:1467] Processing CreateTablet for tablet 59a2ecbd0fbf4bdb81b1548a151648d0 (DEFAULT_TABLE table=test-table [id=14011b244ac04780babaae7c282ac468]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:11:24.151051 24086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59a2ecbd0fbf4bdb81b1548a151648d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:24.152987 24437 tablet_bootstrap.cc:492] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f: Bootstrap starting.
I20260928 13:11:24.153502 24438 tablet_bootstrap.cc:492] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d: Bootstrap starting.
I20260928 13:11:24.154027 24437 tablet_bootstrap.cc:654] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:24.154076 24438 tablet_bootstrap.cc:654] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:24.154292 24439 tablet_bootstrap.cc:492] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Bootstrap starting.
I20260928 13:11:24.154718 24438 log.cc:824] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:24.154775 24437 log.cc:824] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:24.154877 24439 tablet_bootstrap.cc:654] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:24.155460 24439 log.cc:824] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:24.155550 24438 tablet_bootstrap.cc:492] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d: No bootstrap required, opened a new log
I20260928 13:11:24.155607 24438 ts_tablet_manager.cc:1397] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:11:24.156333 24439 tablet_bootstrap.cc:492] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: No bootstrap required, opened a new log
I20260928 13:11:24.156389 24439 ts_tablet_manager.cc:1397] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:11:24.157251 24437 tablet_bootstrap.cc:492] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f: No bootstrap required, opened a new log
I20260928 13:11:24.157332 24437 ts_tablet_manager.cc:1397] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 13:11:24.157603 24438 raft_consensus.cc:348] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:24.157785 24438 raft_consensus.cc:374] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:24.157820 24438 raft_consensus.cc:729] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993e0001b04a455d8a728975e202216d, State: Initialized, Role: FOLLOWER
I20260928 13:11:24.157938 24438 consensus_queue.cc:260] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:24.158228 24438 ts_tablet_manager.cc:1428] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:11:24.158260 24286 heartbeater.cc:503] Master 127.8.88.190:39279 was elected leader, sending a full tablet report...
I20260928 13:11:24.158066 24439 raft_consensus.cc:348] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:24.158612 24439 raft_consensus.cc:374] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:24.158982 24439 raft_consensus.cc:729] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d3d782e890437cb4b1b3d9bc79dccd, State: Initialized, Role: FOLLOWER
I20260928 13:11:24.159255 24439 consensus_queue.cc:260] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:24.159324 24437 raft_consensus.cc:348] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:24.159482 24437 raft_consensus.cc:374] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:24.159539 24437 raft_consensus.cc:729] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3546ec45344433ab0fc892c015be7f, State: Initialized, Role: FOLLOWER
I20260928 13:11:24.159624 24152 heartbeater.cc:503] Master 127.8.88.190:39279 was elected leader, sending a full tablet report...
I20260928 13:11:24.159663 24439 ts_tablet_manager.cc:1428] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:11:24.159651 24437 consensus_queue.cc:260] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:24.159951 24437 ts_tablet_manager.cc:1428] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:11:24.160015 24420 heartbeater.cc:503] Master 127.8.88.190:39279 was elected leader, sending a full tablet report...
W20260928 13:11:24.207480 24287 tablet.cc:2369] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:11:24.268585 24153 tablet.cc:2369] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:11:24.364180 24421 tablet.cc:2369] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:11:24.457595 24443 raft_consensus.cc:482] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:11:24.457800 24443 raft_consensus.cc:504] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:24.458294 24443 leader_election.cc:288] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187), e5d3d782e890437cb4b1b3d9bc79dccd (127.8.88.129:46629)
I20260928 13:11:24.461802 24436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.458573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:55335 (local address 127.8.88.129:46629)
0928 13:11:24.458617 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:24.458619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:24.458993 (+ 374us) server_negotiation.cc:407] Connection header received
0928 13:11:24.459029 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:24.459030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:24.459044 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:24.459116 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:24.459313 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.459944 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.461005 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.461412 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.461475 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.461483 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:24.461670 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 13:11:24.461723 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:11:24.461813 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.458512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:46629 (local address 127.8.88.130:55335)
0928 13:11:24.458829 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:24.458839 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:24.458899 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:24.459128 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:24.459129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:24.459130 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:24.459229 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:24.459230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.460058 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.460059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:24.460903 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:24.460906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.461504 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.461505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:24.461582 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.461587 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:11:24.461644 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 13:11:24.461683 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":174,"thread_start_us":79,"threads_started":1}
I20260928 13:11:24.461882 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.458767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:38059 (local address 127.8.88.131:37187)
0928 13:11:24.458852 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:24.458855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:24.458866 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:11:24.458898 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:24.458899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:24.458911 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:24.458974 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:24.459225 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.459861 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.461029 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.461434 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.461543 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.461552 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:24.461756 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 13:11:24.461799 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 13:11:24.462035 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.458505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:37187 (local address 127.8.88.130:38059)
0928 13:11:24.458635 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:24.458650 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:24.458712 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:24.459025 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:24.459027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:24.459030 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:24.459128 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:24.459130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.459967 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.459969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:24.460862 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:24.460865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.461556 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.461557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:24.461666 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.461670 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:11:24.461732 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 13:11:24.461776 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 13:11:24.462435 24106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a2ecbd0fbf4bdb81b1548a151648d0" candidate_uuid: "993e0001b04a455d8a728975e202216d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" is_pre_election: true
I20260928 13:11:24.462430 24374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a2ecbd0fbf4bdb81b1548a151648d0" candidate_uuid: "993e0001b04a455d8a728975e202216d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3546ec45344433ab0fc892c015be7f" is_pre_election: true
I20260928 13:11:24.462642 24374 raft_consensus.cc:2454] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993e0001b04a455d8a728975e202216d in term 0.
I20260928 13:11:24.462636 24106 raft_consensus.cc:2454] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 993e0001b04a455d8a728975e202216d in term 0.
I20260928 13:11:24.462903 24174 leader_election.cc:302] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3546ec45344433ab0fc892c015be7f, 993e0001b04a455d8a728975e202216d; no voters:
I20260928 13:11:24.463050 24443 raft_consensus.cc:2781] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:11:24.463121 24443 raft_consensus.cc:482] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:11:24.463160 24443 raft_consensus.cc:3037] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:24.463932 24443 raft_consensus.cc:504] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:24.464108 24443 leader_election.cc:288] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [CANDIDATE]: Term 1 election: Requested vote from peers 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187), e5d3d782e890437cb4b1b3d9bc79dccd (127.8.88.129:46629)
I20260928 13:11:24.464296 24374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a2ecbd0fbf4bdb81b1548a151648d0" candidate_uuid: "993e0001b04a455d8a728975e202216d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3546ec45344433ab0fc892c015be7f"
I20260928 13:11:24.464358 24106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59a2ecbd0fbf4bdb81b1548a151648d0" candidate_uuid: "993e0001b04a455d8a728975e202216d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd"
I20260928 13:11:24.464416 24374 raft_consensus.cc:3037] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:24.464424 24106 raft_consensus.cc:3037] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:24.465411 24106 raft_consensus.cc:2454] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993e0001b04a455d8a728975e202216d in term 1.
I20260928 13:11:24.465536 24374 raft_consensus.cc:2454] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 993e0001b04a455d8a728975e202216d in term 1.
I20260928 13:11:24.465646 24173 leader_election.cc:302] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 993e0001b04a455d8a728975e202216d, e5d3d782e890437cb4b1b3d9bc79dccd; no voters:
I20260928 13:11:24.465898 24443 raft_consensus.cc:2781] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:11:24.466086 24443 raft_consensus.cc:686] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [term 1 LEADER]: Becoming Leader. State: Replica: 993e0001b04a455d8a728975e202216d, State: Running, Role: LEADER
I20260928 13:11:24.466212 24443 consensus_queue.cc:237] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:24.467093 23959 catalog_manager.cc:5495] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d reported cstate change: term changed from 0 to 1, leader changed from <none> to 993e0001b04a455d8a728975e202216d (127.8.88.130). New cstate: current_term: 1 leader_uuid: "993e0001b04a455d8a728975e202216d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } health_report { overall_health: UNKNOWN } } }
I20260928 13:11:24.503201 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.500667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.8.88.130:42593)
0928 13:11:24.500776 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:24.500778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:24.500785 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:11:24.500818 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:24.500819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:24.500838 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:24.500959 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:11:24.501104 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.501587 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.502376 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:24.502574 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:24.502648 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:24.502834 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:11:24.502943 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:11:24.503042 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:24.503060 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 13:11:24.503084 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 13:11:24.505084 24374 raft_consensus.cc:1261] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f [term 1 FOLLOWER]: Refusing update from remote peer 993e0001b04a455d8a728975e202216d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:11:24.505093 24106 raft_consensus.cc:1261] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Refusing update from remote peer 993e0001b04a455d8a728975e202216d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:11:24.505455 24443 consensus_queue.cc:1035] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:11:24.505594 24447 consensus_queue.cc:1035] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:11:24.509511 24450 mvcc.cc:205] Tried to move back new op lower bound from 7334302042129924096 to 7334302041974333440. Current Snapshot: MvccSnapshot[applied={T|T < 7334302042129924096 or (T in {7334302042129924096})}]
I20260928 13:11:24.509496 24451 mvcc.cc:205] Tried to move back new op lower bound from 7334302042129924096 to 7334302041974333440. Current Snapshot: MvccSnapshot[applied={T|T < 7334302042129924096 or (T in {7334302042129924096})}]
I20260928 13:11:24.510401 24453 mvcc.cc:205] Tried to move back new op lower bound from 7334302042129924096 to 7334302041974333440. Current Snapshot: MvccSnapshot[applied={T|T < 7334302042129924096 or (T in {7334302042129924096})}]
I20260928 13:11:24.510830 23960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:11:24.510936 23960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:11:24.512434 23960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Sending DeleteTablet for 3 replicas of tablet 59a2ecbd0fbf4bdb81b1548a151648d0
I20260928 13:11:24.512802 24220 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a2ecbd0fbf4bdb81b1548a151648d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:11:24 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:36512
I20260928 13:11:24.512786 24354 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a2ecbd0fbf4bdb81b1548a151648d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:11:24 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:47936
I20260928 13:11:24.513049 24086 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a2ecbd0fbf4bdb81b1548a151648d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:11:24 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:34626
I20260928 13:11:24.513312 24461 tablet_replica.cc:330] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: stopping tablet replica
I20260928 13:11:24.513375 24460 tablet_replica.cc:330] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d: stopping tablet replica
I20260928 13:11:24.513370 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 24023
I20260928 13:11:24.513454 24461 raft_consensus.cc:2229] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:11:24.513499 24460 raft_consensus.cc:2229] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:11:24.513613 24460 raft_consensus.cc:2258] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:11:24.513612 24461 raft_consensus.cc:2258] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:11:24.514205 24460 ts_tablet_manager.cc:1905] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:11:24.516352 24459 tablet_replica.cc:330] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f: stopping tablet replica
I20260928 13:11:24.516503 24459 raft_consensus.cc:2229] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:11:24.516633 24459 raft_consensus.cc:2258] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:11:24.516629 24460 ts_tablet_manager.cc:1918] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:11:24.516739 24460 log.cc:1196] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/59a2ecbd0fbf4bdb81b1548a151648d0
I20260928 13:11:24.517109 24460 ts_tablet_manager.cc:1939] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 993e0001b04a455d8a728975e202216d: Deleting consensus metadata
I20260928 13:11:24.517130 24459 ts_tablet_manager.cc:1905] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:11:24.517617 23948 catalog_manager.cc:4841] TS 993e0001b04a455d8a728975e202216d (127.8.88.130:42593): tablet 59a2ecbd0fbf4bdb81b1548a151648d0 (table test-table [id=14011b244ac04780babaae7c282ac468]) successfully deleted
I20260928 13:11:24.519266 24459 ts_tablet_manager.cc:1918] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:11:24.519357 24459 log.cc:1196] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/59a2ecbd0fbf4bdb81b1548a151648d0
I20260928 13:11:24.519754 24459 ts_tablet_manager.cc:1939] T 59a2ecbd0fbf4bdb81b1548a151648d0 P 4d3546ec45344433ab0fc892c015be7f: Deleting consensus metadata
W20260928 13:11:24.521380 23945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:46629 (error 108)
I20260928 13:11:24.521754 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 24157
I20260928 13:11:24.521571 23946 catalog_manager.cc:4841] TS 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187): tablet 59a2ecbd0fbf4bdb81b1548a151648d0 (table test-table [id=14011b244ac04780babaae7c282ac468]) successfully deleted
I20260928 13:11:24.522959 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:24.522808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:46629 (local address 127.0.0.1:34628)
0928 13:11:24.522874 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:24.522917 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 13:11:24.523379 23945 catalog_manager.cc:4584] TS e5d3d782e890437cb4b1b3d9bc79dccd (127.8.88.129:46629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59a2ecbd0fbf4bdb81b1548a151648d0: Network error: Client connection negotiation failed: client connection to 127.8.88.129:46629: connect: Connection refused (error 111)
I20260928 13:11:24.527057 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 24291
I20260928 13:11:24.532092 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 23929
I20260928 13:11:44.538304 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:39279
--webserver_interface=127.8.88.190
--webserver_port=40423
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:39279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:11:44.621384 24463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:44.621655 24463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:44.621694 24463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:44.622915 24463 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 13:11:44.622988 24463 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 13:11:44.623013 24463 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 13:11:44.623045 24463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:11:44.623076 24463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:44.623108 24463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:44.623135 24463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:11:44.623163 24463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:11:44.624758 24463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:39279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:39279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=40423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:44.625119 24463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:44.625372 24463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:11:44.634991 24463 server_base.cc:1034] running on GCE node
W20260928 13:11:44.634948 24472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:44.634948 24469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:44.634907 24470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:44.635457 24463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:44.635787 24463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:44.636997 24463 hybrid_clock.cc:648] HybridClock initialized: now 1790601104636964 us; error 44 us; skew 500 ppm
Sep 28 13:11:44 dist-test-slave-p1lc krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601104, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:44.646615 24463 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:11:44.647140 24463 webserver.cc:460] Webserver started at http://127.8.88.190:40423/ using document root <none> and password file <none>
I20260928 13:11:44.647403 24463 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:44.647460 24463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:44.649099 24463 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:11:44.649888 24479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:44.650199 24463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:11:44.650327 24463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "13036be9c653434c8519b9fb6d728ed7"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "83189544689d3e20645f43c8313f33c6"
server_key_iv: "4ee62a8ae6c368fb076ce82b8c7db915"
server_key_version: "encryptionkey@0"
I20260928 13:11:44.650820 24463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:44.668941 24463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:44.671132 24463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:44.671299 24463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:44.675704 24463 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:39279
I20260928 13:11:44.675742 24531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:39279 every 8 connection(s)
I20260928 13:11:44.676183 24463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:11:44.677803 24532 sys_catalog.cc:263] Verifying existing consensus state
I20260928 13:11:44.678493 24532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Bootstrap starting.
I20260928 13:11:44.684484 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 24463
I20260928 13:11:44.684779 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:46629
--local_ip_for_outbound_sockets=127.8.88.129
--tserver_master_addrs=127.8.88.190:39279
--webserver_port=46677
--webserver_interface=127.8.88.129
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:11:44.687573 24532 log.cc:824] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:44.689752 24532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:11:44.690075 24532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Bootstrap complete.
I20260928 13:11:44.692445 24532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } }
I20260928 13:11:44.692675 24532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13036be9c653434c8519b9fb6d728ed7, State: Initialized, Role: FOLLOWER
I20260928 13:11:44.692847 24532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } }
I20260928 13:11:44.692968 24532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:11:44.693024 24532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:11:44.693075 24532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 13:11:44.694118 24532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } }
I20260928 13:11:44.694204 24532 leader_election.cc:302] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13036be9c653434c8519b9fb6d728ed7; no voters:
I20260928 13:11:44.694423 24532 leader_election.cc:288] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 13:11:44.694468 24536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 13:11:44.694648 24536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 2 LEADER]: Becoming Leader. State: Replica: 13036be9c653434c8519b9fb6d728ed7, State: Running, Role: LEADER
I20260928 13:11:44.694710 24532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:11:44.694797 24536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } }
I20260928 13:11:44.695108 24536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13036be9c653434c8519b9fb6d728ed7. Latest consensus state: current_term: 2 leader_uuid: "13036be9c653434c8519b9fb6d728ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } } }
I20260928 13:11:44.695113 24537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "13036be9c653434c8519b9fb6d728ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } } }
I20260928 13:11:44.695315 24537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:44.695464 24536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: This master's current role is: LEADER
I20260928 13:11:44.695732 24543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:11:44.696653 24543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:11:44.696841 24543 catalog_manager.cc:1259] Loaded cluster ID: 86a882fa714446208acf1ac8eee87a78
I20260928 13:11:44.697117 24543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:11:44.697940 24543 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:11:44.698132 24543 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Loaded TSK: 0
I20260928 13:11:44.706606 24543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Generated new TSK 1
I20260928 13:11:44.706732 24543 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 13:11:44.786705 24534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:44.786943 24534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:44.786970 24534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:44.788259 24534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:44.788318 24534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:44.788348 24534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:11:44.790014 24534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:46629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=46677
--tserver_master_addrs=127.8.88.190:39279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:44.790285 24534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:44.790546 24534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:44.796116 24562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:44.796178 24560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:44.796144 24559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:44.796572 24534 server_base.cc:1034] running on GCE node
I20260928 13:11:44.796809 24534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:44.797123 24534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:44.798311 24534 hybrid_clock.cc:648] HybridClock initialized: now 1790601104798282 us; error 44 us; skew 500 ppm
Sep 28 13:11:44 dist-test-slave-p1lc krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601104, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:44.801429 24534 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:11:44.802071 24534 webserver.cc:460] Webserver started at http://127.8.88.129:46677/ using document root <none> and password file <none>
I20260928 13:11:44.802397 24534 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:44.802493 24534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:44.804554 24534 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:11:44.805548 24569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:44.805898 24534 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:11:44.806005 24534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "e5d3d782e890437cb4b1b3d9bc79dccd"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "ba1c485a311d596e57f3c6c2cfa67ab2"
server_key_iv: "be819a1282fc24f32e7c05f7f6d1a1ab"
server_key_version: "encryptionkey@0"
I20260928 13:11:44.806490 24534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:44.827867 24534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:44.830619 24534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:44.830792 24534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:44.831140 24534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:44.831759 24576 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 13:11:44.833464 24534 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 13:11:44.833534 24534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 13:11:44.833575 24534 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 13:11:44.834519 24534 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 13:11:44.834569 24534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 13:11:44.834676 24576 tablet_bootstrap.cc:492] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Bootstrap starting.
I20260928 13:11:44.842092 24534 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:46629
I20260928 13:11:44.842235 24683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:46629 every 8 connection(s)
I20260928 13:11:44.842739 24534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:11:44.846226 24576 log.cc:824] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:44.850114 24576 tablet_bootstrap.cc:492] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:11:44.850579 24576 tablet_bootstrap.cc:492] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Bootstrap complete.
I20260928 13:11:44.851018 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 24534
I20260928 13:11:44.851306 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:42593
--local_ip_for_outbound_sockets=127.8.88.130
--tserver_master_addrs=127.8.88.190:39279
--webserver_port=42177
--webserver_interface=127.8.88.130
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:11:44.851923 24576 ts_tablet_manager.cc:1397] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
Sep 28 13:11:44 dist-test-slave-p1lc krb5kdc[23916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601104, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:44.853444 24576 raft_consensus.cc:348] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:44.853811 24576 raft_consensus.cc:729] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d3d782e890437cb4b1b3d9bc79dccd, State: Initialized, Role: FOLLOWER
I20260928 13:11:44.854081 24576 consensus_queue.cc:260] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:44.854429 24576 ts_tablet_manager.cc:1428] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260928 13:11:44.857908 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:44.843590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:41385 (local address 127.8.88.190:39279)
0928 13:11:44.844702 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:44.844712 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:44.844753 (+ 41us) server_negotiation.cc:407] Connection header received
0928 13:11:44.844926 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:44.844934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:44.845001 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:44.845148 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:44.846323 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:44.847913 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:44.849675 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:44.850495 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:44.850626 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:44.853637 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:44.853641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:44.853642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:44.853655 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:44.855828 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:44.856305 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:44.856309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:44.856309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:44.856436 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:44.856726 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:44.856727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:44.856728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:44.857052 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:44.857150 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:44.857557 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 13:11:44.857698 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1050,"thread_start_us":100,"threads_started":1}
I20260928 13:11:44.857935 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:44.843518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:41385)
0928 13:11:44.843811 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:44.843859 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:44.844476 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:44.845222 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:44.845229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:44.845816 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:44.846126 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:44.846135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:44.848073 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:44.848075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:44.849557 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:44.849560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:44.850638 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:44.850640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:44.850838 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:44.851484 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:44.851495 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:44.853490 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:44.855934 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:44.855940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:44.855942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:44.856213 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:44.856526 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:44.856527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:44.856528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:44.856643 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:44.857136 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:44.857145 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:44.857344 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 13:11:44.857484 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 13:11:44.857694 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":111,"threads_started":1}
I20260928 13:11:44.858856 24684 heartbeater.cc:347] Connected to a master server at 127.8.88.190:39279
I20260928 13:11:44.859014 24684 heartbeater.cc:464] Registering TS with master...
I20260928 13:11:44.859256 24684 heartbeater.cc:511] Master 127.8.88.190:39279 requested a full tablet report, sending...
I20260928 13:11:44.860208 24496 ts_manager.cc:194] Registered new tserver with Master: e5d3d782e890437cb4b1b3d9bc79dccd (127.8.88.129:46629)
I20260928 13:11:44.861610 24496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:41385
I20260928 13:11:44.862656 24684 heartbeater.cc:503] Master 127.8.88.190:39279 was elected leader, sending a full tablet report...
I20260928 13:11:44.882676 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:44.861307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:46629 (local address 127.0.0.1:58990)
0928 13:11:44.876660 (+ 15353us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:44.876738 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:44.876924 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:44.877228 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:44.877231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:44.877233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:44.877318 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:44.877336 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:44.880193 (+ 2857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:44.880195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:44.881212 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:44.881216 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:44.882288 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:44.882289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:44.882394 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:44.882408 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 13:11:44.882513 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 13:11:44.882582 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15306,"thread_start_us":76,"threads_started":1}
I20260928 13:11:44.882673 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:44.861082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.8.88.129:46629)
0928 13:11:44.869007 (+ 7925us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:44.869013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:44.876847 (+ 7834us) server_negotiation.cc:407] Connection header received
0928 13:11:44.877074 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:44.877083 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:44.877101 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:44.877220 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:44.877403 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:44.878495 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:44.881311 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:44.882189 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:44.882275 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:44.882345 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:44.882534 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 13:11:44.882582 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7872,"thread_start_us":134,"threads_started":1}
I20260928 13:11:44.883173 24618 tablet_service.cc:1514] Processing DeleteTablet for tablet 59a2ecbd0fbf4bdb81b1548a151648d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:11:24 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:58990
I20260928 13:11:44.883896 24693 tablet_replica.cc:330] stopping tablet replica
I20260928 13:11:44.884034 24693 raft_consensus.cc:2229] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:11:44.884128 24693 raft_consensus.cc:2258] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:11:44.884441 24693 ts_tablet_manager.cc:1905] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:11:44.887302 24693 ts_tablet_manager.cc:1918] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:11:44.887451 24693 log.cc:1196] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal/wals/59a2ecbd0fbf4bdb81b1548a151648d0
I20260928 13:11:44.887707 24693 ts_tablet_manager.cc:1939] T 59a2ecbd0fbf4bdb81b1548a151648d0 P e5d3d782e890437cb4b1b3d9bc79dccd: Deleting consensus metadata
I20260928 13:11:44.888377 24480 catalog_manager.cc:4841] TS e5d3d782e890437cb4b1b3d9bc79dccd (127.8.88.129:46629): tablet 59a2ecbd0fbf4bdb81b1548a151648d0 (table test-table [id=14011b244ac04780babaae7c282ac468]) successfully deleted
W20260928 13:11:44.956851 24689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:44.957120 24689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:44.957168 24689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:44.958950 24689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:44.959018 24689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:44.959050 24689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:11:44.960693 24689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:42593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=42177
--tserver_master_addrs=127.8.88.190:39279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:44.961009 24689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:44.961289 24689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:44.967166 24699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:44.967166 24701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:44.967451 24698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:44.967482 24689 server_base.cc:1034] running on GCE node
I20260928 13:11:44.967767 24689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:44.968117 24689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:44.969301 24689 hybrid_clock.cc:648] HybridClock initialized: now 1790601104969287 us; error 30 us; skew 500 ppm
Sep 28 13:11:44 dist-test-slave-p1lc krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601104, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:44.972286 24689 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:11:44.972838 24689 webserver.cc:460] Webserver started at http://127.8.88.130:42177/ using document root <none> and password file <none>
I20260928 13:11:44.973140 24689 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:44.973230 24689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:44.974869 24689 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:11:44.975601 24708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:44.975817 24689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:11:44.975929 24689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "993e0001b04a455d8a728975e202216d"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "c0a62b67c339270c3479ac705484ed7d"
server_key_iv: "3c3e21116a736bad5a9cd03b319fb46b"
server_key_version: "encryptionkey@0"
I20260928 13:11:44.976339 24689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:44.997712 24689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:45.001107 24689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:45.001502 24689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:45.001832 24689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:45.002345 24689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:11:45.002408 24689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:45.002455 24689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:11:45.002498 24689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:45.010155 24689 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:42593
I20260928 13:11:45.010218 24821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:42593 every 8 connection(s)
I20260928 13:11:45.010630 24689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:11:45.015453 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 24689
I20260928 13:11:45.015820 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:37187
--local_ip_for_outbound_sockets=127.8.88.131
--tserver_master_addrs=127.8.88.190:39279
--webserver_port=44687
--webserver_interface=127.8.88.131
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 13:11:45 dist-test-slave-p1lc krb5kdc[23916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601104, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:45.023334 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.011682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:46457 (local address 127.8.88.190:39279)
0928 13:11:45.011746 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:45.011749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:45.012012 (+ 263us) server_negotiation.cc:407] Connection header received
0928 13:11:45.012535 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:45.012536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:45.012547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:45.012612 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:45.013688 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.014283 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.015542 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.015882 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.015970 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.019757 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:45.019759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:45.019759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:45.019768 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:45.021394 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:45.022032 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:45.022033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:45.022033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:45.022092 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:45.022449 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:45.022449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:45.022450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:45.022563 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:45.022615 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:45.023115 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 13:11:45.023232 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 13:11:45.023638 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.011681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:46457)
0928 13:11:45.011902 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:45.011941 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:45.012433 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:45.012669 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:45.012673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:45.013360 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:45.013571 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:45.013577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.014431 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.014442 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:45.015410 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:45.015413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.016047 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.016049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:45.016249 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.016953 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:45.016964 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:45.019619 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:45.021500 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:45.021511 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:45.021514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:45.021932 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:45.022184 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:45.022185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:45.022186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:45.022371 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:45.022637 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:45.022645 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:45.022870 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 13:11:45.023052 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 13:11:45.023298 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":145,"threads_started":1}
I20260928 13:11:45.024266 24822 heartbeater.cc:347] Connected to a master server at 127.8.88.190:39279
I20260928 13:11:45.024398 24822 heartbeater.cc:464] Registering TS with master...
I20260928 13:11:45.024652 24822 heartbeater.cc:511] Master 127.8.88.190:39279 requested a full tablet report, sending...
I20260928 13:11:45.025224 24496 ts_manager.cc:194] Registered new tserver with Master: 993e0001b04a455d8a728975e202216d (127.8.88.130:42593)
I20260928 13:11:45.025687 24496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:46457
W20260928 13:11:45.106335 24825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:11:45.106556 24825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:11:45.106581 24825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:11:45.107813 24825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:11:45.107873 24825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:11:45.107903 24825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:11:45.109452 24825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:37187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=44687
--tserver_master_addrs=127.8.88.190:39279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:11:45.109704 24825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:11:45.109966 24825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:11:45.115505 24833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:45.115494 24831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:11:45.115494 24830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:11:45.115525 24825 server_base.cc:1034] running on GCE node
I20260928 13:11:45.116009 24825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:11:45.116259 24825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:11:45.117453 24825 hybrid_clock.cc:648] HybridClock initialized: now 1790601105117447 us; error 39 us; skew 500 ppm
Sep 28 13:11:45 dist-test-slave-p1lc krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601105, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:11:45.120100 24825 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:11:45.120599 24825 webserver.cc:460] Webserver started at http://127.8.88.131:44687/ using document root <none> and password file <none>
I20260928 13:11:45.120857 24825 fs_manager.cc:359] Metadata directory not provided
I20260928 13:11:45.121002 24825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:11:45.122509 24825 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:11:45.123207 24840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:45.123427 24825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:11:45.123553 24825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "4d3546ec45344433ab0fc892c015be7f"
format_stamp: "Formatted at 2026-09-28 13:11:24 on dist-test-slave-p1lc"
server_key: "bde0d300255cd76be335c62ab06d69e0"
server_key_iv: "c0a71a52b3f06e7dba3c62d0f70571f6"
server_key_version: "encryptionkey@0"
I20260928 13:11:45.123967 24825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:11:45.137650 24825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:11:45.140198 24825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:11:45.140379 24825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:11:45.140645 24825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:11:45.141031 24825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:11:45.141085 24825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:45.141140 24825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:11:45.141168 24825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:45.147620 24825 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:37187
I20260928 13:11:45.147665 24953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:37187 every 8 connection(s)
I20260928 13:11:45.148058 24825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:11:45.153129 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 24825
Sep 28 13:11:45 dist-test-slave-p1lc krb5kdc[23916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601105, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:11:45.158926 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.148925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:43477 (local address 127.8.88.190:39279)
0928 13:11:45.149001 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:45.149002 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:45.149275 (+ 273us) server_negotiation.cc:407] Connection header received
0928 13:11:45.149794 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:45.149795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:45.149807 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:45.149855 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:45.150858 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.151390 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.152423 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.152614 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.152673 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.155423 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:45.155424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:45.155425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:45.155433 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:45.157304 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:45.157778 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:45.157788 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:45.157788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:45.157857 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:45.158120 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:45.158121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:45.158122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:45.158245 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:45.158324 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:45.158671 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 13:11:45.158799 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 13:11:45.159015 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.148999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:43477)
0928 13:11:45.149152 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:45.149189 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:45.149680 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:45.149956 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:45.149959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:45.150533 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:11:45.150733 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:45.150738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.151481 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.151482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:45.152339 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:45.152342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.152691 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.152692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:45.152829 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.153501 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:11:45.153510 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:11:45.155336 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:11:45.157364 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:45.157374 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:45.157377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:45.157672 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:45.157934 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:11:45.157935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:11:45.157935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:11:45.158049 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:11:45.158325 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:11:45.158333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:11:45.158479 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 13:11:45.158615 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 13:11:45.158812 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260928 13:11:45.159600 24954 heartbeater.cc:347] Connected to a master server at 127.8.88.190:39279
I20260928 13:11:45.159730 24954 heartbeater.cc:464] Registering TS with master...
I20260928 13:11:45.159926 24954 heartbeater.cc:511] Master 127.8.88.190:39279 requested a full tablet report, sending...
I20260928 13:11:45.160422 24496 ts_manager.cc:194] Registered new tserver with Master: 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187)
I20260928 13:11:45.160831 24496 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:43477
I20260928 13:11:45.161140 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.153610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.8.88.190:39279)
0928 13:11:45.153767 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:45.153769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:45.153848 (+ 79us) server_negotiation.cc:407] Connection header received
0928 13:11:45.153970 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:45.153971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:45.153984 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:45.154064 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:45.154915 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.155556 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.156603 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.156948 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.157037 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.158290 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:45.158291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:45.158291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:45.158299 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:45.159947 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:45.160327 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:45.160328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:45.160329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:45.160386 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:45.160587 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:45.160588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:45.160588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:45.160703 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:45.160914 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:45.160953 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 13:11:45.161035 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260928 13:11:45.161958 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:11:45.164825 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.162309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58600 (local address 127.8.88.190:39279)
0928 13:11:45.162366 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:45.162367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:45.162511 (+ 144us) server_negotiation.cc:407] Connection header received
0928 13:11:45.162558 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:45.162558 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:45.162587 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:45.162650 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:11:45.162738 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.163182 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.164029 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.164234 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.164304 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.164515 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:11:45.164565 (+ 50us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 13:11:45.164776 (+ 211us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58600: token expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 13:11:45.164934 24958 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58600: token expired
W20260928 13:11:45.165036 24426 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.88.190:39279)
I20260928 13:11:45.171365 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.165334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.8.88.190:39279)
0928 13:11:45.165368 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:45.165369 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:45.165378 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:11:45.165435 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:45.165436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:45.165443 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:45.165487 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:11:45.166043 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.166507 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.167254 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.167442 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.167501 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.168775 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:11:45.168777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:11:45.168777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:11:45.168784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:11:45.170150 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:45.170542 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:45.170543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:45.170543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:45.170613 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:11:45.170863 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:11:45.170864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:11:45.170864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:11:45.171032 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:11:45.171122 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:45.171220 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 13:11:45.171297 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 13:11:45.172688 24496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:11:45.174459 24496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:11:45.176754 24618 tablet_service.cc:1467] Processing CreateTablet for tablet b1b718574a6e4ca7bdb3eb5d2178d0c4 (DEFAULT_TABLE table=test-table [id=8638467ab7b04420804250a95e04ca25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:11:45.177019 24618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1b718574a6e4ca7bdb3eb5d2178d0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:45.178422 24576 tablet_bootstrap.cc:492] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd: Bootstrap starting.
I20260928 13:11:45.178771 24576 tablet_bootstrap.cc:654] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:45.179384 24576 tablet_bootstrap.cc:492] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd: No bootstrap required, opened a new log
I20260928 13:11:45.179436 24576 ts_tablet_manager.cc:1397] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 13:11:45.179559 24576 raft_consensus.cc:348] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:45.179661 24576 raft_consensus.cc:374] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:45.179705 24576 raft_consensus.cc:729] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d3d782e890437cb4b1b3d9bc79dccd, State: Initialized, Role: FOLLOWER
I20260928 13:11:45.179754 24576 consensus_queue.cc:260] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:45.179855 24576 ts_tablet_manager.cc:1428] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 13:11:45.181553 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.176534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:37187 (local address 127.0.0.1:55836)
0928 13:11:45.176673 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:45.176682 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:45.176718 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:45.177143 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:45.177144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:45.177146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:45.177206 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:45.177207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.178258 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.178259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:45.179588 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:45.179603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.181321 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.181322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:45.181376 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.181381 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:11:45.181431 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 13:11:45.181475 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":104,"threads_started":1}
I20260928 13:11:45.181576 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.176766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.8.88.131:37187)
0928 13:11:45.176989 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:45.176994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:45.177004 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:11:45.177044 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:45.177045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:45.177062 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:45.177146 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:45.177268 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.178174 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.179690 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.180470 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.180566 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.180630 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:45.181452 (+ 822us) server_negotiation.cc:299] Negotiation successful
0928 13:11:45.181498 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":116,"threads_started":1}
I20260928 13:11:45.182426 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.176462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:42593 (local address 127.0.0.1:57310)
0928 13:11:45.176513 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:45.176524 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:45.176563 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:45.178708 (+ 2145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:45.178709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:45.178711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:45.178744 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:45.178745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.180160 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.180160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:45.181223 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:45.181226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.182170 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.182171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:45.182233 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.182237 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:11:45.182313 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 13:11:45.182365 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 13:11:45.182351 24888 tablet_service.cc:1467] Processing CreateTablet for tablet b1b718574a6e4ca7bdb3eb5d2178d0c4 (DEFAULT_TABLE table=test-table [id=8638467ab7b04420804250a95e04ca25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:11:45.182724 24888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1b718574a6e4ca7bdb3eb5d2178d0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:45.182829 24961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.176579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.8.88.130:42593)
0928 13:11:45.178524 (+ 1945us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:45.178529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:45.178542 (+ 13us) server_negotiation.cc:407] Connection header received
0928 13:11:45.178589 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:45.178591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:45.178607 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:45.178774 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:45.178893 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.180054 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.181326 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.182083 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.182149 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.182190 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:45.182442 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 13:11:45.182631 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1902,"thread_start_us":92,"threads_started":1}
I20260928 13:11:45.185756 24964 tablet_bootstrap.cc:492] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f: Bootstrap starting.
I20260928 13:11:45.186424 24964 tablet_bootstrap.cc:654] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:45.186493 24756 tablet_service.cc:1467] Processing CreateTablet for tablet b1b718574a6e4ca7bdb3eb5d2178d0c4 (DEFAULT_TABLE table=test-table [id=8638467ab7b04420804250a95e04ca25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:11:45.186851 24756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1b718574a6e4ca7bdb3eb5d2178d0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:11:45.187068 24964 log.cc:824] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:45.187898 24964 tablet_bootstrap.cc:492] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f: No bootstrap required, opened a new log
I20260928 13:11:45.187960 24964 ts_tablet_manager.cc:1397] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:11:45.189771 24964 raft_consensus.cc:348] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:45.189917 24964 raft_consensus.cc:374] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:45.189939 24964 raft_consensus.cc:729] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3546ec45344433ab0fc892c015be7f, State: Initialized, Role: FOLLOWER
I20260928 13:11:45.190066 24964 consensus_queue.cc:260] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:45.190274 24966 tablet_bootstrap.cc:492] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d: Bootstrap starting.
I20260928 13:11:45.190379 24964 ts_tablet_manager.cc:1428] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:11:45.190515 24954 heartbeater.cc:503] Master 127.8.88.190:39279 was elected leader, sending a full tablet report...
I20260928 13:11:45.190856 24966 tablet_bootstrap.cc:654] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d: Neither blocks nor log segments found. Creating new log.
I20260928 13:11:45.191406 24966 log.cc:824] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d: Log is configured to *not* fsync() on all Append() calls
I20260928 13:11:45.192286 24966 tablet_bootstrap.cc:492] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d: No bootstrap required, opened a new log
I20260928 13:11:45.192344 24966 ts_tablet_manager.cc:1397] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:11:45.193917 24966 raft_consensus.cc:348] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:45.194114 24966 raft_consensus.cc:374] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:11:45.194144 24966 raft_consensus.cc:729] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993e0001b04a455d8a728975e202216d, State: Initialized, Role: FOLLOWER
I20260928 13:11:45.194267 24966 consensus_queue.cc:260] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:45.194517 24966 ts_tablet_manager.cc:1428] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:11:45.194581 24822 heartbeater.cc:503] Master 127.8.88.190:39279 was elected leader, sending a full tablet report...
W20260928 13:11:45.261363 24823 tablet.cc:2369] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:11:45.314520 24690 raft_consensus.cc:482] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:11:45.314781 24690 raft_consensus.cc:504] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:45.315183 24690 leader_election.cc:288] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 993e0001b04a455d8a728975e202216d (127.8.88.130:42593), 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187)
I20260928 13:11:45.318827 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.315393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:42593 (local address 127.8.88.129:35351)
0928 13:11:45.315446 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:45.315459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:45.315513 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:45.315681 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:45.315682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:45.315685 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:45.315755 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:45.315758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.316678 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.316679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:45.317985 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:45.317988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.318551 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.318552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:45.318618 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.318621 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:11:45.318687 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 13:11:45.318732 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 13:11:45.318814 24961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.315470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:35351 (local address 127.8.88.130:42593)
0928 13:11:45.315528 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:45.315530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:45.315541 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:11:45.315574 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:45.315574 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:45.315588 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:45.315661 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:45.315871 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.316584 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.318076 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.318455 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.318542 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.318552 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:45.318714 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 13:11:45.318753 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:11:45.319110 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.315569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:38851 (local address 127.8.88.131:37187)
0928 13:11:45.315616 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:45.315623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:45.315671 (+ 48us) server_negotiation.cc:407] Connection header received
0928 13:11:45.315706 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:45.315707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:45.315716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:45.315771 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:11:45.316355 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.317044 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.318348 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.318737 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.318833 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.318841 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:45.319002 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 13:11:45.319050 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:11:45.319168 24776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1b718574a6e4ca7bdb3eb5d2178d0c4" candidate_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993e0001b04a455d8a728975e202216d" is_pre_election: true
I20260928 13:11:45.319370 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.315467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:37187 (local address 127.8.88.129:38851)
0928 13:11:45.315594 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:45.315603 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:11:45.315644 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:11:45.315854 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:11:45.315856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:11:45.315859 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:11:45.316239 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:45.316249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.317137 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.317138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:45.318246 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:11:45.318248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.318835 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.318836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:11:45.318903 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.318907 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:11:45.319225 (+ 318us) client_negotiation.cc:240] Negotiation successful
0928 13:11:45.319272 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260928 13:11:45.319389 24776 raft_consensus.cc:2454] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5d3d782e890437cb4b1b3d9bc79dccd in term 0.
I20260928 13:11:45.319669 24573 leader_election.cc:302] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 993e0001b04a455d8a728975e202216d, e5d3d782e890437cb4b1b3d9bc79dccd; no voters:
I20260928 13:11:45.319813 24690 raft_consensus.cc:2781] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:11:45.319764 24908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1b718574a6e4ca7bdb3eb5d2178d0c4" candidate_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3546ec45344433ab0fc892c015be7f" is_pre_election: true
I20260928 13:11:45.319880 24690 raft_consensus.cc:482] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:11:45.319909 24690 raft_consensus.cc:3037] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:45.319919 24908 raft_consensus.cc:2454] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5d3d782e890437cb4b1b3d9bc79dccd in term 0.
I20260928 13:11:45.320650 24690 raft_consensus.cc:504] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:45.320815 24690 leader_election.cc:288] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 1 election: Requested vote from peers 993e0001b04a455d8a728975e202216d (127.8.88.130:42593), 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187)
I20260928 13:11:45.320997 24908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1b718574a6e4ca7bdb3eb5d2178d0c4" candidate_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3546ec45344433ab0fc892c015be7f"
I20260928 13:11:45.321026 24776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1b718574a6e4ca7bdb3eb5d2178d0c4" candidate_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "993e0001b04a455d8a728975e202216d"
I20260928 13:11:45.321087 24908 raft_consensus.cc:3037] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:45.321110 24776 raft_consensus.cc:3037] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:11:45.321805 24776 raft_consensus.cc:2454] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5d3d782e890437cb4b1b3d9bc79dccd in term 1.
I20260928 13:11:45.321853 24908 raft_consensus.cc:2454] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5d3d782e890437cb4b1b3d9bc79dccd in term 1.
I20260928 13:11:45.322000 24573 leader_election.cc:302] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 993e0001b04a455d8a728975e202216d, e5d3d782e890437cb4b1b3d9bc79dccd; no voters:
I20260928 13:11:45.322091 24690 raft_consensus.cc:2781] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:11:45.322281 24690 raft_consensus.cc:686] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 LEADER]: Becoming Leader. State: Replica: e5d3d782e890437cb4b1b3d9bc79dccd, State: Running, Role: LEADER
I20260928 13:11:45.322342 24690 consensus_queue.cc:237] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:11:45.323194 24495 catalog_manager.cc:5495] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd reported cstate change: term changed from 0 to 1, leader changed from <none> to e5d3d782e890437cb4b1b3d9bc79dccd (127.8.88.129). New cstate: current_term: 1 leader_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } health_report { overall_health: HEALTHY } } }
W20260928 13:11:45.343659 24685 tablet.cc:2369] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:11:45.365324 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.362543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.8.88.129:46629)
0928 13:11:45.362637 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 13:11:45.362639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:11:45.362650 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:11:45.362691 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:11:45.362692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:11:45.362715 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:11:45.362813 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:11:45.362975 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.363469 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.364367 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:11:45.364561 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:11:45.364618 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:11:45.364849 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:11:45.364982 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:11:45.365044 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 13:11:45.365163 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 13:11:45.365184 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 13:11:45.367223 24908 raft_consensus.cc:1261] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f [term 1 FOLLOWER]: Refusing update from remote peer e5d3d782e890437cb4b1b3d9bc79dccd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:11:45.367260 24776 raft_consensus.cc:1261] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d [term 1 FOLLOWER]: Refusing update from remote peer e5d3d782e890437cb4b1b3d9bc79dccd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:11:45.367700 24971 consensus_queue.cc:1035] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [LEADER]: Connected to new peer: Peer: permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:11:45.367858 24690 consensus_queue.cc:1035] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:11:45.370357 24974 mvcc.cc:205] Tried to move back new op lower bound from 7334302127581507584 to 7334302127401070592. Current Snapshot: MvccSnapshot[applied={T|T < 7334302127581507584}]
I20260928 13:11:45.370774 24975 mvcc.cc:205] Tried to move back new op lower bound from 7334302127581507584 to 7334302127401070592. Current Snapshot: MvccSnapshot[applied={T|T < 7334302127581507584}]
I20260928 13:11:45.370831 24976 mvcc.cc:205] Tried to move back new op lower bound from 7334302127581507584 to 7334302127401070592. Current Snapshot: MvccSnapshot[applied={T|T < 7334302127581507584}]
I20260928 13:11:45.372854 24496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:11:45.372993 24496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:11:45.373924 24496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Sending DeleteTablet for 3 replicas of tablet b1b718574a6e4ca7bdb3eb5d2178d0c4
I20260928 13:11:45.374293 24888 tablet_service.cc:1514] Processing DeleteTablet for tablet b1b718574a6e4ca7bdb3eb5d2178d0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:11:45 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:55836
I20260928 13:11:45.374368 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 24534
I20260928 13:11:45.374552 24756 tablet_service.cc:1514] Processing DeleteTablet for tablet b1b718574a6e4ca7bdb3eb5d2178d0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:11:45 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:57310
I20260928 13:11:45.375027 24985 tablet_replica.cc:330] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d: stopping tablet replica
I20260928 13:11:45.375133 24985 raft_consensus.cc:2229] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:11:45.375241 24985 raft_consensus.cc:2258] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:11:45.375687 24985 ts_tablet_manager.cc:1905] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:11:45.375895 24984 tablet_replica.cc:330] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f: stopping tablet replica
I20260928 13:11:45.376015 24984 raft_consensus.cc:2229] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:11:45.376117 24984 raft_consensus.cc:2258] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:11:45.376665 24984 ts_tablet_manager.cc:1905] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:11:45.377919 24985 ts_tablet_manager.cc:1918] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:11:45.378028 24985 log.cc:1196] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/b1b718574a6e4ca7bdb3eb5d2178d0c4
I20260928 13:11:45.378356 24985 ts_tablet_manager.cc:1939] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 993e0001b04a455d8a728975e202216d: Deleting consensus metadata
I20260928 13:11:45.378679 24984 ts_tablet_manager.cc:1918] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:11:45.378753 24984 log.cc:1196] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/b1b718574a6e4ca7bdb3eb5d2178d0c4
I20260928 13:11:45.378834 24483 catalog_manager.cc:4841] TS 993e0001b04a455d8a728975e202216d (127.8.88.130:42593): tablet b1b718574a6e4ca7bdb3eb5d2178d0c4 (table test-table [id=8638467ab7b04420804250a95e04ca25]) successfully deleted
I20260928 13:11:45.379011 24984 ts_tablet_manager.cc:1939] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P 4d3546ec45344433ab0fc892c015be7f: Deleting consensus metadata
I20260928 13:11:45.379484 24481 catalog_manager.cc:4841] TS 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187): tablet b1b718574a6e4ca7bdb3eb5d2178d0c4 (table test-table [id=8638467ab7b04420804250a95e04ca25]) successfully deleted
W20260928 13:11:45.380899 24480 connection.cc:535] client connection to 127.8.88.129:46629 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 13:11:45.381019 24480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 13:11:45.381188 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 24689
I20260928 13:11:45.381990 24692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:11:45.381844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:46629 (local address 127.0.0.1:59010)
0928 13:11:45.381895 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 13:11:45.381950 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 13:11:45.382115 24480 catalog_manager.cc:4584] TS e5d3d782e890437cb4b1b3d9bc79dccd (127.8.88.129:46629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1b718574a6e4ca7bdb3eb5d2178d0c4: Network error: Client connection negotiation failed: client connection to 127.8.88.129:46629: connect: Connection refused (error 111)
I20260928 13:11:45.387549 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 24825
I20260928 13:11:45.392967 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 24463
I20260928 13:12:05.398938 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:39279
--webserver_interface=127.8.88.190
--webserver_port=40423
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:39279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:05.493891 24987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:05.494191 24987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:05.494220 24987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:05.495656 24987 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 13:12:05.495711 24987 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 13:12:05.495725 24987 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 13:12:05.495738 24987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:05.495752 24987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:05.495767 24987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:05.495780 24987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:05.495843 24987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:05.497746 24987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:39279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:39279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=40423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:05.498116 24987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:05.498448 24987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:05.505165 24995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:05.505178 24993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:05.505215 24987 server_base.cc:1034] running on GCE node
W20260928 13:12:05.505178 24992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:05.505771 24987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:05.506170 24987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:05.507369 24987 hybrid_clock.cc:648] HybridClock initialized: now 1790601125507336 us; error 50 us; skew 500 ppm
Sep 28 13:12:05 dist-test-slave-p1lc krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601125, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:05.521660 24987 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:12:05.522527 24987 webserver.cc:460] Webserver started at http://127.8.88.190:40423/ using document root <none> and password file <none>
I20260928 13:12:05.522940 24987 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:05.523041 24987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:05.525441 24987 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:05.526459 25002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:05.526721 24987 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:05.526875 24987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "13036be9c653434c8519b9fb6d728ed7"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "83189544689d3e20645f43c8313f33c6"
server_key_iv: "4ee62a8ae6c368fb076ce82b8c7db915"
server_key_version: "encryptionkey@0"
I20260928 13:12:05.527554 24987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:05.542804 24987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:05.546913 24987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:05.547143 24987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:05.554286 24987 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:39279
I20260928 13:12:05.554306 25054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:39279 every 8 connection(s)
I20260928 13:12:05.555004 24987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:05.555783 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 24987
I20260928 13:12:05.556143 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:46629
--local_ip_for_outbound_sockets=127.8.88.129
--tserver_master_addrs=127.8.88.190:39279
--webserver_port=46677
--webserver_interface=127.8.88.129
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:12:05.557137 25055 sys_catalog.cc:263] Verifying existing consensus state
I20260928 13:12:05.558482 25055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Bootstrap starting.
I20260928 13:12:05.573148 25055 log.cc:824] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:05.576241 25055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:12:05.576671 25055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: Bootstrap complete.
I20260928 13:12:05.579972 25055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } }
I20260928 13:12:05.580294 25055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13036be9c653434c8519b9fb6d728ed7, State: Initialized, Role: FOLLOWER
I20260928 13:12:05.580514 25055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } }
I20260928 13:12:05.580591 25055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:05.580652 25055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:05.580711 25055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 2 FOLLOWER]: Advancing to term 3
I20260928 13:12:05.582080 25055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } }
I20260928 13:12:05.582240 25055 leader_election.cc:302] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13036be9c653434c8519b9fb6d728ed7; no voters:
I20260928 13:12:05.582479 25055 leader_election.cc:288] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 13:12:05.582526 25059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 13:12:05.582659 25059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [term 3 LEADER]: Becoming Leader. State: Replica: 13036be9c653434c8519b9fb6d728ed7, State: Running, Role: LEADER
I20260928 13:12:05.582758 25059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } }
I20260928 13:12:05.582808 25055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:05.583078 25059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13036be9c653434c8519b9fb6d728ed7. Latest consensus state: current_term: 3 leader_uuid: "13036be9c653434c8519b9fb6d728ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } } }
I20260928 13:12:05.583155 25059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:05.583086 25060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "13036be9c653434c8519b9fb6d728ed7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13036be9c653434c8519b9fb6d728ed7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39279 } } }
I20260928 13:12:05.583432 25060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:05.584807 25074 catalog_manager.cc:1259] Loaded cluster ID: 86a882fa714446208acf1ac8eee87a78
I20260928 13:12:05.584923 25074 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: loading cluster ID for follower catalog manager: success
I20260928 13:12:05.585784 25074 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 13036be9c653434c8519b9fb6d728ed7: acquiring CA information for follower catalog manager: success
F20260928 13:12:05.586185 25074 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcdc96f79ad google::LogMessage::Fail() at ??:0
@ 0x7fcdc96fa88a google::LogMessage::SendToLog() at ??:0
@ 0x7fcdc96f74de google::LogMessage::Flush() at ??:0
@ 0x7fcdc96f8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcdcbe64104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fcdd16506b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fcdd1650989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fcdd1650e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fcdd16538c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fcdcb5ba954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcdcc41e6db start_thread at ??:0
@ 0x7fcdc81f561f clone at ??:0
W20260928 13:12:05.690483 25057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:05.690784 25057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:05.690829 25057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:05.692611 25057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:05.692679 25057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:05.692723 25057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:12:05.694957 25057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:46629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=46677
--tserver_master_addrs=127.8.88.190:39279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:05.695255 25057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:05.695554 25057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:05.701378 25093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:05.701538 25057 server_base.cc:1034] running on GCE node
W20260928 13:12:05.701385 25094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:05.701586 25096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:05.701828 25057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:05.702121 25057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:05.703310 25057 hybrid_clock.cc:648] HybridClock initialized: now 1790601125703291 us; error 32 us; skew 500 ppm
Sep 28 13:12:05 dist-test-slave-p1lc krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601125, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:05.706650 25057 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:12:05.707247 25057 webserver.cc:460] Webserver started at http://127.8.88.129:46677/ using document root <none> and password file <none>
I20260928 13:12:05.707526 25057 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:05.707599 25057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:05.709667 25057 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:12:05.710491 25103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:05.710709 25057 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:05.710829 25057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "e5d3d782e890437cb4b1b3d9bc79dccd"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "ba1c485a311d596e57f3c6c2cfa67ab2"
server_key_iv: "be819a1282fc24f32e7c05f7f6d1a1ab"
server_key_version: "encryptionkey@0"
I20260928 13:12:05.711333 25057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:05.747277 25057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:05.749766 25057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:05.749922 25057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:05.750191 25057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:05.750665 25110 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 13:12:05.752710 25057 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 13:12:05.752779 25057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 13:12:05.752818 25057 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 13:12:05.753537 25057 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 13:12:05.753598 25057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 13:12:05.753669 25110 tablet_bootstrap.cc:492] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd: Bootstrap starting.
I20260928 13:12:05.761059 25057 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:46629
I20260928 13:12:05.761209 25217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:46629 every 8 connection(s)
I20260928 13:12:05.761615 25057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:12:05.764705 25110 log.cc:824] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:05.766052 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 25057
I20260928 13:12:05.766372 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:42593
--local_ip_for_outbound_sockets=127.8.88.130
--tserver_master_addrs=127.8.88.190:39279
--webserver_port=42177
--webserver_interface=127.8.88.130
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:12:05.767624 25110 tablet_bootstrap.cc:492] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 13:12:05.767971 25110 tablet_bootstrap.cc:492] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd: Bootstrap complete.
I20260928 13:12:05.769258 25110 ts_tablet_manager.cc:1397] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.015s
I20260928 13:12:05.770730 25110 raft_consensus.cc:348] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:12:05.771051 25110 raft_consensus.cc:729] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d3d782e890437cb4b1b3d9bc79dccd, State: Initialized, Role: FOLLOWER
I20260928 13:12:05.771221 25110 consensus_queue.cc:260] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:12:05.771510 25110 ts_tablet_manager.cc:1428] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260928 13:12:05.815410 25220 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:05.762306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:51141)
0928 13:12:05.762484 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:05.762528 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:05.763006 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:05.815192 (+ 52186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
W20260928 13:12:05.815734 25218 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:39279 (0 consecutive failures): Network error: Failed to ping master at 127.8.88.190:39279: Client connection negotiation failed: client connection to 127.8.88.190:39279: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 13:12:05.816166 25220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:05.816036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:53613)
0928 13:12:05.816092 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:05.816140 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 13:12:05.816509 25220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:05.816423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:47393)
0928 13:12:05.816466 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:05.816492 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 13:12:05.816646 25218 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 13:12:05.854640 25222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:05.854863 25222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:05.854889 25222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:05.856138 25222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:05.856184 25222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:05.856212 25222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:12:05.857776 25222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:42593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=42177
--tserver_master_addrs=127.8.88.190:39279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:05.858042 25222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:05.858302 25222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:05.864007 25231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:05.864082 25228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:05.864169 25222 server_base.cc:1034] running on GCE node
W20260928 13:12:05.864082 25229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:05.864507 25222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:05.864773 25222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:05.865959 25222 hybrid_clock.cc:648] HybridClock initialized: now 1790601125865924 us; error 48 us; skew 500 ppm
Sep 28 13:12:05 dist-test-slave-p1lc krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601125, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:05.869016 25222 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:12:05.869578 25222 webserver.cc:460] Webserver started at http://127.8.88.130:42177/ using document root <none> and password file <none>
I20260928 13:12:05.869823 25222 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:05.869894 25222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:05.871347 25222 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:05.872133 25238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:05.872380 25222 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:05.872502 25222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "993e0001b04a455d8a728975e202216d"
format_stamp: "Formatted at 2026-09-28 13:11:23 on dist-test-slave-p1lc"
server_key: "c0a62b67c339270c3479ac705484ed7d"
server_key_iv: "3c3e21116a736bad5a9cd03b319fb46b"
server_key_version: "encryptionkey@0"
I20260928 13:12:05.872882 25222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:05.896283 25222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:05.899169 25222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:05.899426 25222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:05.899703 25222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:05.900077 25222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:05.900135 25222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:05.900173 25222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:05.900216 25222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:05.906847 25222 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:42593
I20260928 13:12:05.907090 25351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:42593 every 8 connection(s)
I20260928 13:12:05.907359 25222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:12:05.908591 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:05.908197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:35223)
0928 13:12:05.908363 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:05.908439 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
W20260928 13:12:05.909096 25352 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:39279 (0 consecutive failures): Network error: Failed to ping master at 127.8.88.190:39279: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
I20260928 13:12:05.909430 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:05.909315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:55615)
0928 13:12:05.909374 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:05.909407 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 13:12:05.909792 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:05.909717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:44863)
0928 13:12:05.909754 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:05.909776 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 13:12:05.909920 25352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 13:12:05.913146 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 25222
I20260928 13:12:05.913411 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:37187
--local_ip_for_outbound_sockets=127.8.88.131
--tserver_master_addrs=127.8.88.190:39279
--webserver_port=44687
--webserver_interface=127.8.88.131
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:05.999923 25355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:06.000147 25355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:06.000173 25355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:06.001474 25355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:06.001526 25355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:06.001555 25355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:12:06.003046 25355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:37187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=44687
--tserver_master_addrs=127.8.88.190:39279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:06.003244 25355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:06.003434 25355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:06.009087 25360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:06.009087 25363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:06.009087 25361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:06.009119 25355 server_base.cc:1034] running on GCE node
I20260928 13:12:06.009620 25355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:06.009961 25355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:06.011183 25355 hybrid_clock.cc:648] HybridClock initialized: now 1790601126011166 us; error 32 us; skew 500 ppm
Sep 28 13:12:06 dist-test-slave-p1lc krb5kdc[23916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601126, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:06.014209 25355 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:12:06.014721 25355 webserver.cc:460] Webserver started at http://127.8.88.131:44687/ using document root <none> and password file <none>
I20260928 13:12:06.014972 25355 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:06.015046 25355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:06.016604 25355 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:06.017426 25370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:06.017643 25355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:12:06.017753 25355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "4d3546ec45344433ab0fc892c015be7f"
format_stamp: "Formatted at 2026-09-28 13:11:24 on dist-test-slave-p1lc"
server_key: "bde0d300255cd76be335c62ab06d69e0"
server_key_iv: "c0a71a52b3f06e7dba3c62d0f70571f6"
server_key_version: "encryptionkey@0"
I20260928 13:12:06.018287 25355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:06.037433 25355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:06.040011 25355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:06.040191 25355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:06.040452 25355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:06.040800 25355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:06.040856 25355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:06.041042 25355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:06.041070 25355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:06.047284 25355 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:37187
I20260928 13:12:06.047330 25483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:37187 every 8 connection(s)
I20260928 13:12:06.047708 25355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:12:06.048818 25486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:06.048394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:38543)
0928 13:12:06.048574 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:06.048656 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":67,"threads_started":1}
I20260928 13:12:06.049190 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 25355
W20260928 13:12:06.049377 25484 heartbeater.cc:644] Failed to heartbeat to 127.8.88.190:39279 (0 consecutive failures): Network error: Failed to ping master at 127.8.88.190:39279: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
I20260928 13:12:06.049721 25486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:06.049592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:51883)
0928 13:12:06.049651 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:06.049686 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 13:12:06.050117 25486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:06.050036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:39865)
0928 13:12:06.050066 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:06.050091 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260928 13:12:06.050218 25484 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 13:12:06.817291 25488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:06.817048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:44239)
0928 13:12:06.817181 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:06.817223 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260928 13:12:06.910601 25489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:06.910328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:37833)
0928 13:12:06.910481 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:06.910528 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260928 13:12:07.050880 25490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:07.050622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:49355)
0928 13:12:07.050778 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:07.050823 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":59,"threads_started":1}
I20260928 13:12:07.174934 25491 raft_consensus.cc:482] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:12:07.175076 25491 raft_consensus.cc:504] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:12:07.175465 25491 leader_election.cc:288] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 993e0001b04a455d8a728975e202216d (127.8.88.130:42593), 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187)
Sep 28 13:12:07 dist-test-slave-p1lc krb5kdc[23916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601125, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.130@KRBTEST.COM
Sep 28 13:12:07 dist-test-slave-p1lc krb5kdc[23916](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601125, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.131@KRBTEST.COM
I20260928 13:12:07.187448 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:07.175831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:48639 (local address 127.8.88.130:42593)
0928 13:12:07.175996 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:07.176007 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:07.176043 (+ 36us) server_negotiation.cc:407] Connection header received
0928 13:12:07.176511 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:07.176513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:07.176557 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:07.176711 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:07.177607 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:07.178497 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:07.179985 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:07.180735 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:07.180895 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:07.183601 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:07.183606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:07.183606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:07.183620 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:07.185625 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:07.186130 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:07.186133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:07.186134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:07.186234 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:07.186519 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:07.186520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:07.186520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:07.186785 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:07.186889 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:07.187182 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 13:12:07.187324 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260928 13:12:07.187443 25493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:07.175697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:42593 (local address 127.8.88.129:48639)
0928 13:12:07.175959 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:07.175971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:07.176031 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:07.176753 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:07.176755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:07.177257 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:07.177465 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:07.177471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:07.178626 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:07.178627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:07.179833 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:07.179836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:07.180919 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:07.180921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:07.181129 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:07.181756 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:07.181777 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:07.183468 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:07.185741 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:07.185750 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:07.185752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:07.186037 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:07.186315 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:07.186316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:07.186316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:07.186446 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:07.186862 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:07.186870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:07.187025 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 13:12:07.187119 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 13:12:07.187339 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":188,"thread_start_us":70,"threads_started":1}
I20260928 13:12:07.187477 25492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:07.175791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:33893 (local address 127.8.88.131:37187)
0928 13:12:07.175971 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:07.175983 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:07.176017 (+ 34us) server_negotiation.cc:407] Connection header received
0928 13:12:07.176531 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:07.176533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:07.176572 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:07.176716 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:07.178667 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:07.179995 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:07.181017 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:07.181596 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:07.181716 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:07.183839 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:07.183843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:07.183843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:07.183853 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:07.185704 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:07.186169 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:07.186172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:07.186172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:07.186269 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:07.186522 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:07.186523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:07.186523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:07.186780 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:07.186887 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:07.187200 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 13:12:07.187356 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":59,"threads_started":1}
I20260928 13:12:07.187845 25488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:07.175682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:37187 (local address 127.8.88.129:33893)
0928 13:12:07.175739 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:07.175752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:07.175825 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:07.176757 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:07.176758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:07.177258 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:07.177465 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:07.177471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:07.180094 (+ 2623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:07.180095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:07.180934 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:07.180937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:07.181712 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:07.181715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:07.181768 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:07.182246 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:07.182250 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:07.183721 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:07.185820 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:07.185821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:07.185822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:07.186069 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:07.186394 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:07.186395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:07.186395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:07.186454 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:07.186878 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:07.186878 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:07.187007 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 13:12:07.187139 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 13:12:07.187425 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":111}
W20260928 13:12:07.188514 25107 leader_election.cc:341] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 993e0001b04a455d8a728975e202216d (127.8.88.130:42593): Not found: Tablet not found: b1b718574a6e4ca7bdb3eb5d2178d0c4
W20260928 13:12:07.188716 25105 leader_election.cc:341] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187): Not found: Tablet not found: b1b718574a6e4ca7bdb3eb5d2178d0c4
I20260928 13:12:07.188793 25105 leader_election.cc:302] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5d3d782e890437cb4b1b3d9bc79dccd; no voters: 4d3546ec45344433ab0fc892c015be7f, 993e0001b04a455d8a728975e202216d
I20260928 13:12:07.188969 25491 raft_consensus.cc:2726] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:12:07.818261 25495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:07.817981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:51569)
0928 13:12:07.818139 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:07.818187 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
I20260928 13:12:07.911420 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:07.911178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:52399)
0928 13:12:07.911327 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:07.911372 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260928 13:12:08.051755 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:08.051500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:48425)
0928 13:12:08.051660 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:08.051706 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20260928 13:12:08.819015 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:08.818770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:35777)
0928 13:12:08.818913 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:08.818961 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260928 13:12:08.912310 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:08.912054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:44661)
0928 13:12:08.912208 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:08.912253 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":65,"threads_started":1}
I20260928 13:12:09.052647 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:09.052389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:42015)
0928 13:12:09.052556 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:09.052601 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260928 13:12:09.347275 25501 raft_consensus.cc:482] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:12:09.347397 25501 raft_consensus.cc:504] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:12:09.347666 25501 leader_election.cc:288] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 993e0001b04a455d8a728975e202216d (127.8.88.130:42593), 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187)
W20260928 13:12:09.348121 25107 leader_election.cc:341] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 993e0001b04a455d8a728975e202216d (127.8.88.130:42593): Not found: Tablet not found: b1b718574a6e4ca7bdb3eb5d2178d0c4
W20260928 13:12:09.348209 25105 leader_election.cc:341] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187): Not found: Tablet not found: b1b718574a6e4ca7bdb3eb5d2178d0c4
I20260928 13:12:09.348307 25105 leader_election.cc:302] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5d3d782e890437cb4b1b3d9bc79dccd; no voters: 4d3546ec45344433ab0fc892c015be7f, 993e0001b04a455d8a728975e202216d
I20260928 13:12:09.348389 25501 raft_consensus.cc:2726] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:12:09.819972 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:09.819690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:49167)
0928 13:12:09.819860 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:09.819907 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260928 13:12:09.913307 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:09.913032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:55183)
0928 13:12:09.913220 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:09.913259 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":62,"threads_started":1}
I20260928 13:12:10.053651 25504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:10.053405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:55437)
0928 13:12:10.053567 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:10.053606 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
I20260928 13:12:10.821014 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:10.820649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:51919)
0928 13:12:10.820812 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:10.820915 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260928 13:12:10.914409 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:10.913961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:52387)
0928 13:12:10.914282 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:10.914331 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":126,"threads_started":1}
I20260928 13:12:11.054450 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:11.054173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:47397)
0928 13:12:11.054364 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:11.054401 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":103,"threads_started":1}
I20260928 13:12:11.821799 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:11.821551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:45783)
0928 13:12:11.821710 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:11.821754 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":70,"threads_started":1}
I20260928 13:12:11.915256 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:11.914987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:60857)
0928 13:12:11.915143 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:11.915192 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260928 13:12:12.055270 25510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:12.055027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:56307)
0928 13:12:12.055181 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:12.055223 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260928 13:12:12.488726 25511 raft_consensus.cc:482] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:12:12.488832 25511 raft_consensus.cc:504] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:12:12.489171 25511 leader_election.cc:288] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 993e0001b04a455d8a728975e202216d (127.8.88.130:42593), 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187)
W20260928 13:12:12.489521 25107 leader_election.cc:341] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 993e0001b04a455d8a728975e202216d (127.8.88.130:42593): Not found: Tablet not found: b1b718574a6e4ca7bdb3eb5d2178d0c4
W20260928 13:12:12.489626 25105 leader_election.cc:341] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187): Not found: Tablet not found: b1b718574a6e4ca7bdb3eb5d2178d0c4
I20260928 13:12:12.489658 25107 leader_election.cc:302] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5d3d782e890437cb4b1b3d9bc79dccd; no voters: 4d3546ec45344433ab0fc892c015be7f, 993e0001b04a455d8a728975e202216d
I20260928 13:12:12.489776 25511 raft_consensus.cc:2726] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:12:12.822826 25512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:12.822481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:49983)
0928 13:12:12.822701 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:12.822758 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":105,"threads_started":1}
I20260928 13:12:12.916296 25513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:12.915964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:37577)
0928 13:12:12.916171 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:12.916229 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":121,"threads_started":1}
I20260928 13:12:13.056574 25514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:13.056171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:42577)
0928 13:12:13.056415 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:13.056473 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":121,"threads_started":1}
I20260928 13:12:13.823645 25515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:13.823403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:58351)
0928 13:12:13.823545 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:13.823588 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260928 13:12:13.917271 25516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:13.917029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:37929)
0928 13:12:13.917173 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:13.917219 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260928 13:12:14.057616 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:14.057359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:42255)
0928 13:12:14.057520 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:14.057567 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260928 13:12:14.824446 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:14.824228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:40475)
0928 13:12:14.824359 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:14.824397 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260928 13:12:14.918079 25519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:14.917811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:43521)
0928 13:12:14.917966 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:14.918017 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260928 13:12:15.058693 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:15.058308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:32861)
0928 13:12:15.058576 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:15.058626 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":79,"threads_started":1}
I20260928 13:12:15.192507 25521 raft_consensus.cc:482] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:12:15.192616 25521 raft_consensus.cc:504] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "993e0001b04a455d8a728975e202216d" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 42593 } } peers { permanent_uuid: "4d3546ec45344433ab0fc892c015be7f" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 37187 } } peers { permanent_uuid: "e5d3d782e890437cb4b1b3d9bc79dccd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 46629 } }
I20260928 13:12:15.192835 25521 leader_election.cc:288] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 993e0001b04a455d8a728975e202216d (127.8.88.130:42593), 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187)
W20260928 13:12:15.193262 25107 leader_election.cc:341] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 993e0001b04a455d8a728975e202216d (127.8.88.130:42593): Not found: Tablet not found: b1b718574a6e4ca7bdb3eb5d2178d0c4
W20260928 13:12:15.193348 25105 leader_election.cc:341] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d3546ec45344433ab0fc892c015be7f (127.8.88.131:37187): Not found: Tablet not found: b1b718574a6e4ca7bdb3eb5d2178d0c4
I20260928 13:12:15.193379 25107 leader_election.cc:302] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5d3d782e890437cb4b1b3d9bc79dccd; no voters: 4d3546ec45344433ab0fc892c015be7f, 993e0001b04a455d8a728975e202216d
I20260928 13:12:15.193507 25521 raft_consensus.cc:2726] T b1b718574a6e4ca7bdb3eb5d2178d0c4 P e5d3d782e890437cb4b1b3d9bc79dccd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 13:12:15.825232 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:15.824998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:36663)
0928 13:12:15.825150 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:15.825187 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260928 13:12:15.918900 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:15.918658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:42097)
0928 13:12:15.918804 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:15.918851 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260928 13:12:16.059468 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:16.059236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:52613)
0928 13:12:16.059386 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:16.059422 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260928 13:12:16.826058 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:16.825767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:46301)
0928 13:12:16.825927 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:16.825975 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260928 13:12:16.919781 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:16.919568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:48311)
0928 13:12:16.919696 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:16.919734 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260928 13:12:17.060485 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:17.060204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:48031)
0928 13:12:17.060388 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:17.060437 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260928 13:12:17.826807 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:17.826587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:40415)
0928 13:12:17.826716 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:17.826759 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260928 13:12:17.920585 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:17.920327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:41127)
0928 13:12:17.920464 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:17.920509 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260928 13:12:18.061296 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:18.061054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:58383)
0928 13:12:18.061190 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:18.061237 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260928 13:12:18.827741 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:18.827488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:43239)
0928 13:12:18.827643 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:18.827690 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260928 13:12:18.921489 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:18.921182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:57263)
0928 13:12:18.921393 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:18.921439 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":65,"threads_started":1}
I20260928 13:12:19.062194 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:19.061920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:36609)
0928 13:12:19.062093 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:19.062148 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260928 13:12:19.828593 25534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:19.828313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:46321)
0928 13:12:19.828485 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:19.828542 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260928 13:12:19.922214 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:19.921971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:40499)
0928 13:12:19.922112 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:19.922162 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260928 13:12:20.063187 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:20.062882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:52833)
0928 13:12:20.063085 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:20.063136 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
I20260928 13:12:20.829509 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:20.829268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.129:41395)
0928 13:12:20.829409 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:20.829461 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260928 13:12:20.923036 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:20.922783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.130:36327)
0928 13:12:20.922943 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:20.922987 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":77,"threads_started":1}
I20260928 13:12:21.057039 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 25057
I20260928 13:12:21.062237 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 25222
I20260928 13:12:21.064234 25539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:21.063846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39279 (local address 127.8.88.131:53085)
0928 13:12:21.063977 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:21.064021 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.190:39279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260928 13:12:21.067126 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 25355
2026-09-28T13:12:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58793 ms)
[----------] 30 tests from SecurityITest (266615 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:21 dist-test-slave-p1lc krb5kdc[25542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:21 dist-test-slave-p1lc krb5kdc[25542](info): set up 2 sockets
Sep 28 13:12:21 dist-test-slave-p1lc krb5kdc[25542](info): commencing operation
krb5kdc: starting...
W20260928 13:12:22.140743 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:22 dist-test-slave-p1lc krb5kdc[25542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:12:22.269275 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:44263
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:44263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:22.352327 25555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:22.352571 25555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:22.352595 25555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:22.353866 25555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:22.353920 25555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:22.353935 25555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:22.353950 25555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:22.353969 25555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:22.355568 25555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:44263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:44263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:22.355860 25555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:22.356128 25555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:22.361874 25561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:22.361847 25560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:22.362076 25555 server_base.cc:1034] running on GCE node
W20260928 13:12:22.361857 25563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:22.362347 25555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:22.362695 25555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:22.363886 25555 hybrid_clock.cc:648] HybridClock initialized: now 1790601142363862 us; error 41 us; skew 500 ppm
Sep 28 13:12:22 dist-test-slave-p1lc krb5kdc[25542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601142, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:22.366514 25555 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:12:22.367027 25555 webserver.cc:460] Webserver started at http://127.8.88.190:46075/ using document root <none> and password file <none>
I20260928 13:12:22.367274 25555 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:22.367345 25555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:22.367493 25555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:22.368571 25555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "d38f9af9e9e84ca9a961db0af6bead89"
format_stamp: "Formatted at 2026-09-28 13:12:22 on dist-test-slave-p1lc"
server_key: "378d210c08b3f2ec9ecff00be5581580"
server_key_iv: "ef09920b276dedb3600c6064b94aae6d"
server_key_version: "encryptionkey@0"
I20260928 13:12:22.369002 25555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "d38f9af9e9e84ca9a961db0af6bead89"
format_stamp: "Formatted at 2026-09-28 13:12:22 on dist-test-slave-p1lc"
server_key: "378d210c08b3f2ec9ecff00be5581580"
server_key_iv: "ef09920b276dedb3600c6064b94aae6d"
server_key_version: "encryptionkey@0"
I20260928 13:12:22.370383 25555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:22.371282 25570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:22.371563 25555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:22.371695 25555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "d38f9af9e9e84ca9a961db0af6bead89"
format_stamp: "Formatted at 2026-09-28 13:12:22 on dist-test-slave-p1lc"
server_key: "378d210c08b3f2ec9ecff00be5581580"
server_key_iv: "ef09920b276dedb3600c6064b94aae6d"
server_key_version: "encryptionkey@0"
I20260928 13:12:22.371779 25555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:22.393745 25555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:22.396075 25555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:22.396260 25555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:22.400622 25555 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:44263
I20260928 13:12:22.400660 25622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:44263 every 8 connection(s)
I20260928 13:12:22.401132 25555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:22.402071 25623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:22.404343 25623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89: Bootstrap starting.
I20260928 13:12:22.404641 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 25555
I20260928 13:12:22.404760 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:22.405002 8546 external_mini_cluster.cc:1428] Setting key 1da70b262299d8c6b4e5da21cf723faa
I20260928 13:12:22.405174 25623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:22.405575 25623 log.cc:824] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:22.406436 25623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89: No bootstrap required, opened a new log
I20260928 13:12:22.408420 25623 raft_consensus.cc:348] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38f9af9e9e84ca9a961db0af6bead89" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44263 } }
I20260928 13:12:22.408632 25623 raft_consensus.cc:374] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:22.408671 25623 raft_consensus.cc:729] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d38f9af9e9e84ca9a961db0af6bead89, State: Initialized, Role: FOLLOWER
I20260928 13:12:22.408810 25623 consensus_queue.cc:260] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38f9af9e9e84ca9a961db0af6bead89" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44263 } }
I20260928 13:12:22.408934 25623 raft_consensus.cc:388] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:22.408977 25623 raft_consensus.cc:482] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:22.409025 25623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:22.409618 25623 raft_consensus.cc:504] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38f9af9e9e84ca9a961db0af6bead89" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44263 } }
I20260928 13:12:22.409708 25623 leader_election.cc:302] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d38f9af9e9e84ca9a961db0af6bead89; no voters:
I20260928 13:12:22.409919 25623 leader_election.cc:288] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:22.409964 25628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:22.410125 25628 raft_consensus.cc:686] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [term 1 LEADER]: Becoming Leader. State: Replica: d38f9af9e9e84ca9a961db0af6bead89, State: Running, Role: LEADER
I20260928 13:12:22.410283 25623 sys_catalog.cc:564] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:22.410260 25628 consensus_queue.cc:237] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38f9af9e9e84ca9a961db0af6bead89" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44263 } }
I20260928 13:12:22.410775 25629 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d38f9af9e9e84ca9a961db0af6bead89. Latest consensus state: current_term: 1 leader_uuid: "d38f9af9e9e84ca9a961db0af6bead89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38f9af9e9e84ca9a961db0af6bead89" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44263 } } }
I20260928 13:12:22.410871 25629 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:22.411119 25630 sys_catalog.cc:455] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d38f9af9e9e84ca9a961db0af6bead89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d38f9af9e9e84ca9a961db0af6bead89" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44263 } } }
I20260928 13:12:22.411453 25630 sys_catalog.cc:458] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:22.411652 25636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 13:12:22 dist-test-slave-p1lc krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:22.412269 25636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:22.414093 25636 catalog_manager.cc:1347] Generated new cluster ID: d9090ca0361a4c27a73f2ac61a0e79b3
I20260928 13:12:22.414151 25636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:22.416774 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.405613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.8.88.190:44263)
0928 13:12:22.405789 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:22.405798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:22.405833 (+ 35us) server_negotiation.cc:407] Connection header received
0928 13:12:22.406396 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:22.406404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:22.406463 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:22.406642 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:22.407281 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.408212 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.409316 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.409831 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.409936 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.412802 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:22.412806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:22.412807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:22.412820 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:22.414946 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:22.415467 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:22.415470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:22.415471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:22.415580 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:22.415802 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:22.415803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:22.415803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:22.416088 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:22.416229 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:22.416425 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 13:12:22.416565 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260928 13:12:22.432060 25636 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:22.432647 25636 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:22.438141 25636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89: Generated new TSK 0
I20260928 13:12:22.438351 25636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:12:22.472214 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44263
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:22.557331 25649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:22.557616 25649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:22.557663 25649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:22.558962 25649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:22.559058 25649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:22.559109 25649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:12:22.560760 25649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:22.561085 25649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:22.561339 25649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:22.566870 25654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:22.566887 25655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:22.566887 25657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:22.567211 25649 server_base.cc:1034] running on GCE node
I20260928 13:12:22.567451 25649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:22.567768 25649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:22.568970 25649 hybrid_clock.cc:648] HybridClock initialized: now 1790601142568915 us; error 66 us; skew 500 ppm
Sep 28 13:12:22 dist-test-slave-p1lc krb5kdc[25542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601142, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:22.571888 25649 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:12:22.572384 25649 webserver.cc:460] Webserver started at http://127.8.88.129:45907/ using document root <none> and password file <none>
I20260928 13:12:22.572620 25649 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:22.572669 25649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:22.572839 25649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:22.574095 25649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "356d19a9ed554b258f5fd468779161cd"
format_stamp: "Formatted at 2026-09-28 13:12:22 on dist-test-slave-p1lc"
server_key: "8a6030de635692c151e77bd4693ee651"
server_key_iv: "c081d2b01bbfb6bb77726033790f7923"
server_key_version: "encryptionkey@0"
I20260928 13:12:22.574492 25649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "356d19a9ed554b258f5fd468779161cd"
format_stamp: "Formatted at 2026-09-28 13:12:22 on dist-test-slave-p1lc"
server_key: "8a6030de635692c151e77bd4693ee651"
server_key_iv: "c081d2b01bbfb6bb77726033790f7923"
server_key_version: "encryptionkey@0"
I20260928 13:12:22.576177 25649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:12:22.577242 25664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:22.577517 25649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:12:22.577618 25649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "356d19a9ed554b258f5fd468779161cd"
format_stamp: "Formatted at 2026-09-28 13:12:22 on dist-test-slave-p1lc"
server_key: "8a6030de635692c151e77bd4693ee651"
server_key_iv: "c081d2b01bbfb6bb77726033790f7923"
server_key_version: "encryptionkey@0"
I20260928 13:12:22.577692 25649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:22.592262 25649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:22.595191 25649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:22.595374 25649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:22.595611 25649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:22.595984 25649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:22.596047 25649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:22.596086 25649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:22.596122 25649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:22.602972 25649 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:44495
I20260928 13:12:22.603020 25777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:44495 every 8 connection(s)
I20260928 13:12:22.603389 25649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:12:22.607983 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 25649
I20260928 13:12:22.608119 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:12:22.608359 8546 external_mini_cluster.cc:1428] Setting key a04a1af4497cb8eb7bcd51fe4314cc7b
Sep 28 13:12:22 dist-test-slave-p1lc krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601142, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:22.615177 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.604192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:55135 (local address 127.8.88.190:44263)
0928 13:12:22.604273 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:22.604275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:22.604527 (+ 252us) server_negotiation.cc:407] Connection header received
0928 13:12:22.605157 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:22.605159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:22.605172 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:22.605232 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:22.606319 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.606892 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.608192 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.608570 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.608627 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.611550 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:22.611552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:22.611552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:22.611562 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:22.613537 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:22.613974 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:22.613976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:22.613976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:22.614050 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:22.614328 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:22.614329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:22.614329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:22.614509 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:22.614607 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:22.614910 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 13:12:22.615016 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 13:12:22.615195 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.604190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:44263 (local address 127.8.88.129:55135)
0928 13:12:22.604386 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:22.604429 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:22.605016 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:22.605312 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:22.605315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:22.605965 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:22.606192 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:22.606198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.607005 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.607006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:22.608059 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:22.608063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.608730 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.608731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:22.608848 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.609638 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:22.609647 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:22.611441 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:22.613649 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:22.613657 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:22.613659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:22.613890 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:22.614129 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:22.614130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:22.614130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:22.614245 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:22.614612 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:22.614618 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:22.614757 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 13:12:22.614855 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 13:12:22.614997 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":100,"threads_started":1}
I20260928 13:12:22.615800 25778 heartbeater.cc:347] Connected to a master server at 127.8.88.190:44263
I20260928 13:12:22.615919 25778 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:22.616150 25778 heartbeater.cc:511] Master 127.8.88.190:44263 requested a full tablet report, sending...
I20260928 13:12:22.616755 25587 ts_manager.cc:194] Registered new tserver with Master: 356d19a9ed554b258f5fd468779161cd (127.8.88.129:44495)
I20260928 13:12:22.617444 25587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:55135
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:12:22.637190 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44263
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:22.719937 25783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:22.720170 25783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:22.720196 25783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:22.721688 25783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:22.721750 25783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:22.721779 25783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:12:22.723407 25783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:22.723673 25783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:22.723941 25783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:22.729449 25789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:22.729454 25791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:22.729454 25788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:22.729707 25783 server_base.cc:1034] running on GCE node
I20260928 13:12:22.730003 25783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:22.730261 25783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:22.731452 25783 hybrid_clock.cc:648] HybridClock initialized: now 1790601142731428 us; error 39 us; skew 500 ppm
Sep 28 13:12:22 dist-test-slave-p1lc krb5kdc[25542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601142, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:22.734494 25783 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:12:22.735028 25783 webserver.cc:460] Webserver started at http://127.8.88.130:33931/ using document root <none> and password file <none>
I20260928 13:12:22.735320 25783 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:22.735392 25783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:22.735545 25783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:22.736650 25783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "b51e3e44c56f47a7b4f2417db01129e3"
format_stamp: "Formatted at 2026-09-28 13:12:22 on dist-test-slave-p1lc"
server_key: "28387a43bd3e38c23fc7026b5519069b"
server_key_iv: "1a3e885f027ffca40eddc10cae333824"
server_key_version: "encryptionkey@0"
I20260928 13:12:22.737111 25783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "b51e3e44c56f47a7b4f2417db01129e3"
format_stamp: "Formatted at 2026-09-28 13:12:22 on dist-test-slave-p1lc"
server_key: "28387a43bd3e38c23fc7026b5519069b"
server_key_iv: "1a3e885f027ffca40eddc10cae333824"
server_key_version: "encryptionkey@0"
I20260928 13:12:22.738582 25783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:12:22.739462 25798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:22.739733 25783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:22.739835 25783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "b51e3e44c56f47a7b4f2417db01129e3"
format_stamp: "Formatted at 2026-09-28 13:12:22 on dist-test-slave-p1lc"
server_key: "28387a43bd3e38c23fc7026b5519069b"
server_key_iv: "1a3e885f027ffca40eddc10cae333824"
server_key_version: "encryptionkey@0"
I20260928 13:12:22.739919 25783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:22.750245 25783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:22.753062 25783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:22.753248 25783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:22.753515 25783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:22.753907 25783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:22.753968 25783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:22.754004 25783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:22.754042 25783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:22.760454 25783 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:38447
I20260928 13:12:22.760499 25911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:38447 every 8 connection(s)
I20260928 13:12:22.760921 25783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:12:22.762358 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 25783
I20260928 13:12:22.762496 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:12:22.762717 8546 external_mini_cluster.cc:1428] Setting key 02125069971412e815ed28417f332cb1
Sep 28 13:12:22 dist-test-slave-p1lc krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601142, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
I20260928 13:12:22.773178 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.761863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:37119 (local address 127.8.88.190:44263)
0928 13:12:22.761935 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:22.761938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:22.762248 (+ 310us) server_negotiation.cc:407] Connection header received
0928 13:12:22.763088 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:22.763089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:22.763143 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:22.763312 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:22.764733 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.765331 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.766410 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.766704 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.766788 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.769802 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:22.769804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:22.769805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:22.769814 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:22.771559 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:22.771975 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:22.771977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:22.771977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:22.772043 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:22.772306 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:22.772307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:22.772308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:22.772467 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:22.772650 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:22.772984 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 13:12:22.773073 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 13:12:22.773408 25914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.761853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:44263 (local address 127.8.88.130:37119)
0928 13:12:22.762052 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:22.762098 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:22.762961 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:22.763396 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:22.763400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:22.764313 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:22.764626 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:22.764632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.765440 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.765441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:22.766312 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:22.766314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.766794 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.766795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:22.766935 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.767573 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:22.767584 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:22.769651 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:22.771645 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:22.771653 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:22.771656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:22.771895 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:22.772112 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:22.772113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:22.772114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:22.772236 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:22.772635 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:22.772643 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:22.772801 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 13:12:22.772959 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 13:12:22.773149 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":104,"threads_started":1}
I20260928 13:12:22.773882 25912 heartbeater.cc:347] Connected to a master server at 127.8.88.190:44263
I20260928 13:12:22.774006 25912 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
I20260928 13:12:22.774269 25912 heartbeater.cc:511] Master 127.8.88.190:44263 requested a full tablet report, sending...
I20260928 13:12:22.774798 25587 ts_manager.cc:194] Registered new tserver with Master: b51e3e44c56f47a7b4f2417db01129e3 (127.8.88.130:38447)
I20260928 13:12:22.775247 25587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:37119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:12:22.792465 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44263
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:22.879439 25917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:22.879666 25917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:22.879691 25917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:22.881026 25917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:22.881074 25917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:22.881102 25917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:12:22.882563 25917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:22.882761 25917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:22.882943 25917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:22.889061 25925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:22.889078 25923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:22.889084 25922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:22.889353 25917 server_base.cc:1034] running on GCE node
I20260928 13:12:22.889707 25917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:22.889955 25917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:22.891147 25917 hybrid_clock.cc:648] HybridClock initialized: now 1790601142891138 us; error 37 us; skew 500 ppm
Sep 28 13:12:22 dist-test-slave-p1lc krb5kdc[25542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601142, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:22.894271 25917 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:12:22.894881 25917 webserver.cc:460] Webserver started at http://127.8.88.131:33659/ using document root <none> and password file <none>
I20260928 13:12:22.895151 25917 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:22.895232 25917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:22.895399 25917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:22.896674 25917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "c1136fd063984589a33050dc861d3f52"
format_stamp: "Formatted at 2026-09-28 13:12:22 on dist-test-slave-p1lc"
server_key: "ebcf1d42d498fbffeafc3e0fce8e8d0d"
server_key_iv: "8962c33af9e5c9f84b107e1fd92092d6"
server_key_version: "encryptionkey@0"
I20260928 13:12:22.897110 25917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "c1136fd063984589a33050dc861d3f52"
format_stamp: "Formatted at 2026-09-28 13:12:22 on dist-test-slave-p1lc"
server_key: "ebcf1d42d498fbffeafc3e0fce8e8d0d"
server_key_iv: "8962c33af9e5c9f84b107e1fd92092d6"
server_key_version: "encryptionkey@0"
I20260928 13:12:22.898800 25917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:12:22.899856 25932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:22.900220 25917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:22.900349 25917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "c1136fd063984589a33050dc861d3f52"
format_stamp: "Formatted at 2026-09-28 13:12:22 on dist-test-slave-p1lc"
server_key: "ebcf1d42d498fbffeafc3e0fce8e8d0d"
server_key_iv: "8962c33af9e5c9f84b107e1fd92092d6"
server_key_version: "encryptionkey@0"
I20260928 13:12:22.900437 25917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:22.919960 25917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:22.922806 25917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:22.923022 25917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:22.923308 25917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:22.923699 25917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:22.923763 25917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:22.923849 25917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:22.923884 25917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:22.931455 25917 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:40167
I20260928 13:12:22.931639 26045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:40167 every 8 connection(s)
I20260928 13:12:22.932011 25917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:12:22.938584 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 25917
I20260928 13:12:22.938717 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:12:22.938921 8546 external_mini_cluster.cc:1428] Setting key c1e53768feb2d1d5c0d61425e4a4a727
Sep 28 13:12:22 dist-test-slave-p1lc krb5kdc[25542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601142, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:22.942768 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.932935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:46339 (local address 127.8.88.190:44263)
0928 13:12:22.932994 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:22.932997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:22.933250 (+ 253us) server_negotiation.cc:407] Connection header received
0928 13:12:22.933839 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:22.933840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:22.933855 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:22.933908 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:22.934934 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.935462 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.936545 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.936750 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.936839 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.939542 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:22.939544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:22.939545 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:22.939554 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:22.941014 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:22.941455 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:22.941456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:22.941456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:22.941597 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:22.941942 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:22.941943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:22.941944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:22.942098 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:22.942172 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:22.942566 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 13:12:22.942694 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:12:22.942852 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.932926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:44263 (local address 127.8.88.131:46339)
0928 13:12:22.933115 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:22.933154 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:22.933713 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:22.933996 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:22.934000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:22.934604 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:22.934822 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:22.934828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.935576 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.935577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:22.936429 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:22.936432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.936855 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.936856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:22.937040 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.937670 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:22.937679 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:22.939436 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:22.941108 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:22.941116 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:22.941118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:22.941372 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:22.941705 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:22.941706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:22.941706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:22.941840 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:22.942199 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:22.942206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:22.942349 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 13:12:22.942452 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 13:12:22.942620 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
I20260928 13:12:22.943411 26046 heartbeater.cc:347] Connected to a master server at 127.8.88.190:44263
I20260928 13:12:22.943518 26046 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:22.943750 26046 heartbeater.cc:511] Master 127.8.88.190:44263 requested a full tablet report, sending...
I20260928 13:12:22.944275 25587 ts_manager.cc:194] Registered new tserver with Master: c1136fd063984589a33050dc861d3f52 (127.8.88.131:40167)
I20260928 13:12:22.944770 25587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:46339
I20260928 13:12:22.951082 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:12:22.959410 25626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.952763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.8.88.190:44263)
0928 13:12:22.952821 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:22.952823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:22.953056 (+ 233us) server_negotiation.cc:407] Connection header received
0928 13:12:22.953095 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:22.953096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:22.953106 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:22.953147 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:22.953828 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.954336 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.955139 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.955330 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.955392 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.956732 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:22.956734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:22.956734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:22.956741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:22.958373 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:22.958723 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:22.958724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:22.958725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:22.958775 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:22.958946 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:22.958947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:22.958947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:22.959093 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:22.959181 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:22.959281 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 13:12:22.959342 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 13:12:22.961091 25587 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:12:22.962862 25587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:12:22.970220 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.965505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.8.88.130:38447)
0928 13:12:22.966492 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:22.966496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:22.966505 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:12:22.966544 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:22.966547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:22.966562 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:22.966630 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:22.966852 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.967867 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.969097 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.969857 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.969938 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.969994 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:22.970084 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 13:12:22.970126 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":129,"threads_started":1}
I20260928 13:12:22.970150 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.965410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:38447 (local address 127.0.0.1:49880)
0928 13:12:22.965669 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:22.965716 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:22.965791 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:22.966711 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:22.966712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:22.966714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:22.966767 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:22.966769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.967967 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.967968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:22.968992 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:22.968995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.969944 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.969945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:22.970005 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.970009 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:12:22.970057 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 13:12:22.970089 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":3,"thread_start_us":66,"threads_started":1}
I20260928 13:12:22.970424 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.965411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:44495 (local address 127.0.0.1:52138)
0928 13:12:22.965712 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:22.965721 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:22.965791 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:22.966207 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:22.966209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:22.966211 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:22.966295 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:22.966297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.967706 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.967707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:22.969058 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:22.969059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.970179 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.970179 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:22.970236 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.970240 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:12:22.970292 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 13:12:22.970333 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":11,"thread_start_us":188,"threads_started":1}
I20260928 13:12:22.970732 26062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.965766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.8.88.129:44495)
0928 13:12:22.965950 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:22.965954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:22.965965 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:12:22.966009 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:22.966012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:22.966049 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:22.966139 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:22.966388 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.967567 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.969129 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.969856 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.969938 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.969994 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:22.970584 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 13:12:22.970638 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
I20260928 13:12:22.970943 26061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.965744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.8.88.131:40167)
0928 13:12:22.965948 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:22.965962 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:22.966415 (+ 453us) server_negotiation.cc:407] Connection header received
0928 13:12:22.966471 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:22.966473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:22.966490 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:22.966570 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:22.966710 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.967445 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.969160 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.969671 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.969766 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.969833 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:22.970251 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 13:12:22.970861 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":77,"threads_started":1}
I20260928 13:12:22.971117 25846 tablet_service.cc:1467] Processing CreateTablet for tablet 5605cef16e3444c2a02685cac2e4712c (DEFAULT_TABLE table=test-table [id=874b4120c0854447981a6224b5edcd38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:22.971498 25846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5605cef16e3444c2a02685cac2e4712c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:22.971439 25712 tablet_service.cc:1467] Processing CreateTablet for tablet 5605cef16e3444c2a02685cac2e4712c (DEFAULT_TABLE table=test-table [id=874b4120c0854447981a6224b5edcd38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:22.971756 25712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5605cef16e3444c2a02685cac2e4712c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:22.974596 26063 tablet_bootstrap.cc:492] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3: Bootstrap starting.
I20260928 13:12:22.974598 26064 tablet_bootstrap.cc:492] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd: Bootstrap starting.
I20260928 13:12:22.970150 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:22.965411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:40167 (local address 127.0.0.1:46658)
0928 13:12:22.966270 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:22.966279 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:22.966353 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:22.966583 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:22.966584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:22.966586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:22.966628 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:22.966629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.967550 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.967552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:22.969056 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:22.969059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:22.969768 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:22.969769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:22.969854 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:22.969868 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 13:12:22.969978 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 13:12:22.970049 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"mutex_wait_us":77,"thread_start_us":182,"threads_started":1}
I20260928 13:12:22.975406 26063 tablet_bootstrap.cc:654] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:22.975425 26064 tablet_bootstrap.cc:654] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:22.975805 25980 tablet_service.cc:1467] Processing CreateTablet for tablet 5605cef16e3444c2a02685cac2e4712c (DEFAULT_TABLE table=test-table [id=874b4120c0854447981a6224b5edcd38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:22.975982 26063 log.cc:824] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:22.976006 26064 log.cc:824] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:22.976276 25980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5605cef16e3444c2a02685cac2e4712c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:22.976979 26064 tablet_bootstrap.cc:492] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd: No bootstrap required, opened a new log
I20260928 13:12:22.977053 26064 ts_tablet_manager.cc:1397] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 13:12:22.977090 26063 tablet_bootstrap.cc:492] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3: No bootstrap required, opened a new log
I20260928 13:12:22.977161 26063 ts_tablet_manager.cc:1397] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 13:12:22.978698 26063 raft_consensus.cc:348] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356d19a9ed554b258f5fd468779161cd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44495 } } peers { permanent_uuid: "b51e3e44c56f47a7b4f2417db01129e3" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38447 } } peers { permanent_uuid: "c1136fd063984589a33050dc861d3f52" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40167 } }
I20260928 13:12:22.978881 26063 raft_consensus.cc:374] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:22.978928 26063 raft_consensus.cc:729] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51e3e44c56f47a7b4f2417db01129e3, State: Initialized, Role: FOLLOWER
I20260928 13:12:22.979075 26063 consensus_queue.cc:260] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356d19a9ed554b258f5fd468779161cd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44495 } } peers { permanent_uuid: "b51e3e44c56f47a7b4f2417db01129e3" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38447 } } peers { permanent_uuid: "c1136fd063984589a33050dc861d3f52" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40167 } }
I20260928 13:12:22.979413 26063 ts_tablet_manager.cc:1428] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:12:22.979543 25912 heartbeater.cc:503] Master 127.8.88.190:44263 was elected leader, sending a full tablet report...
I20260928 13:12:22.979468 26064 raft_consensus.cc:348] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356d19a9ed554b258f5fd468779161cd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44495 } } peers { permanent_uuid: "b51e3e44c56f47a7b4f2417db01129e3" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38447 } } peers { permanent_uuid: "c1136fd063984589a33050dc861d3f52" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40167 } }
I20260928 13:12:22.979660 26064 raft_consensus.cc:374] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:22.979705 26064 raft_consensus.cc:729] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356d19a9ed554b258f5fd468779161cd, State: Initialized, Role: FOLLOWER
I20260928 13:12:22.980140 26068 tablet_bootstrap.cc:492] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52: Bootstrap starting.
I20260928 13:12:22.979823 26064 consensus_queue.cc:260] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356d19a9ed554b258f5fd468779161cd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44495 } } peers { permanent_uuid: "b51e3e44c56f47a7b4f2417db01129e3" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38447 } } peers { permanent_uuid: "c1136fd063984589a33050dc861d3f52" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40167 } }
I20260928 13:12:22.980515 26064 ts_tablet_manager.cc:1428] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:12:22.980542 25778 heartbeater.cc:503] Master 127.8.88.190:44263 was elected leader, sending a full tablet report...
I20260928 13:12:22.981351 26068 tablet_bootstrap.cc:654] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:22.981918 26068 log.cc:824] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:22.982833 26068 tablet_bootstrap.cc:492] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52: No bootstrap required, opened a new log
I20260928 13:12:22.982949 26068 ts_tablet_manager.cc:1397] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 13:12:22.984443 26068 raft_consensus.cc:348] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356d19a9ed554b258f5fd468779161cd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44495 } } peers { permanent_uuid: "b51e3e44c56f47a7b4f2417db01129e3" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38447 } } peers { permanent_uuid: "c1136fd063984589a33050dc861d3f52" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40167 } }
I20260928 13:12:22.984638 26068 raft_consensus.cc:374] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:22.984686 26068 raft_consensus.cc:729] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1136fd063984589a33050dc861d3f52, State: Initialized, Role: FOLLOWER
I20260928 13:12:22.984818 26068 consensus_queue.cc:260] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356d19a9ed554b258f5fd468779161cd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44495 } } peers { permanent_uuid: "b51e3e44c56f47a7b4f2417db01129e3" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38447 } } peers { permanent_uuid: "c1136fd063984589a33050dc861d3f52" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40167 } }
I20260928 13:12:22.985200 26068 ts_tablet_manager.cc:1428] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:12:22.985227 26046 heartbeater.cc:503] Master 127.8.88.190:44263 was elected leader, sending a full tablet report...
W20260928 13:12:23.011662 25913 tablet.cc:2369] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:12:23.094599 26069 raft_consensus.cc:482] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:12:23.094735 26069 raft_consensus.cc:504] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356d19a9ed554b258f5fd468779161cd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44495 } } peers { permanent_uuid: "b51e3e44c56f47a7b4f2417db01129e3" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38447 } } peers { permanent_uuid: "c1136fd063984589a33050dc861d3f52" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40167 } }
I20260928 13:12:23.095057 26069 leader_election.cc:288] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b51e3e44c56f47a7b4f2417db01129e3 (127.8.88.130:38447), c1136fd063984589a33050dc861d3f52 (127.8.88.131:40167)
I20260928 13:12:23.098640 26061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.095506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:54947 (local address 127.8.88.131:40167)
0928 13:12:23.095584 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.095586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.095597 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:12:23.095630 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.095632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.095642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.095721 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:23.095903 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.096516 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.097775 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.098232 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.098295 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.098303 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.098524 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.098567 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 13:12:23.098666 25780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.095287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:40167 (local address 127.8.88.129:54947)
0928 13:12:23.095458 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:23.095471 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:23.095538 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:23.095731 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:23.095732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:23.095734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:23.095826 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.095828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.096601 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.096603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.097667 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.097670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.098328 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.098330 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.098415 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.098420 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:12:23.098495 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 13:12:23.098546 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":71}
I20260928 13:12:23.098903 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.095406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:49061 (local address 127.8.88.130:38447)
0928 13:12:23.095462 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.095464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.095708 (+ 244us) server_negotiation.cc:407] Connection header received
0928 13:12:23.095739 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.095739 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.095758 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.095816 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:23.096063 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.096735 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.097940 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.098429 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.098506 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.098514 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.098790 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.098838 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 13:12:23.099144 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.095287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:38447 (local address 127.8.88.129:49061)
0928 13:12:23.095507 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:23.095543 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:23.095681 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:23.095835 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:23.095836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:23.095837 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:23.095948 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.095949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.096821 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.096822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.097833 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.097836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.098528 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.098530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.098679 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.098685 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 13:12:23.098992 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 13:12:23.099049 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":61,"thread_start_us":92,"threads_started":1}
I20260928 13:12:23.099048 26000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5605cef16e3444c2a02685cac2e4712c" candidate_uuid: "356d19a9ed554b258f5fd468779161cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1136fd063984589a33050dc861d3f52" is_pre_election: true
I20260928 13:12:23.099332 26000 raft_consensus.cc:2454] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 356d19a9ed554b258f5fd468779161cd in term 0.
I20260928 13:12:23.099581 25866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5605cef16e3444c2a02685cac2e4712c" candidate_uuid: "356d19a9ed554b258f5fd468779161cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51e3e44c56f47a7b4f2417db01129e3" is_pre_election: true
I20260928 13:12:23.099633 25666 leader_election.cc:302] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 356d19a9ed554b258f5fd468779161cd, c1136fd063984589a33050dc861d3f52; no voters:
I20260928 13:12:23.099736 25866 raft_consensus.cc:2454] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 356d19a9ed554b258f5fd468779161cd in term 0.
I20260928 13:12:23.099767 26069 raft_consensus.cc:2781] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:12:23.099851 26069 raft_consensus.cc:482] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:12:23.099887 26069 raft_consensus.cc:3037] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:23.100618 26069 raft_consensus.cc:504] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356d19a9ed554b258f5fd468779161cd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44495 } } peers { permanent_uuid: "b51e3e44c56f47a7b4f2417db01129e3" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38447 } } peers { permanent_uuid: "c1136fd063984589a33050dc861d3f52" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40167 } }
I20260928 13:12:23.100771 26069 leader_election.cc:288] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [CANDIDATE]: Term 1 election: Requested vote from peers b51e3e44c56f47a7b4f2417db01129e3 (127.8.88.130:38447), c1136fd063984589a33050dc861d3f52 (127.8.88.131:40167)
I20260928 13:12:23.100988 25866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5605cef16e3444c2a02685cac2e4712c" candidate_uuid: "356d19a9ed554b258f5fd468779161cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b51e3e44c56f47a7b4f2417db01129e3"
I20260928 13:12:23.101025 26000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5605cef16e3444c2a02685cac2e4712c" candidate_uuid: "356d19a9ed554b258f5fd468779161cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1136fd063984589a33050dc861d3f52"
I20260928 13:12:23.101068 25866 raft_consensus.cc:3037] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:23.101136 26000 raft_consensus.cc:3037] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:23.101945 26000 raft_consensus.cc:2454] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 356d19a9ed554b258f5fd468779161cd in term 1.
I20260928 13:12:23.101950 25866 raft_consensus.cc:2454] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 356d19a9ed554b258f5fd468779161cd in term 1.
I20260928 13:12:23.102186 25666 leader_election.cc:302] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 356d19a9ed554b258f5fd468779161cd, c1136fd063984589a33050dc861d3f52; no voters:
I20260928 13:12:23.102303 26069 raft_consensus.cc:2781] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:23.102458 26069 raft_consensus.cc:686] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [term 1 LEADER]: Becoming Leader. State: Replica: 356d19a9ed554b258f5fd468779161cd, State: Running, Role: LEADER
I20260928 13:12:23.102686 26069 consensus_queue.cc:237] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356d19a9ed554b258f5fd468779161cd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44495 } } peers { permanent_uuid: "b51e3e44c56f47a7b4f2417db01129e3" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38447 } } peers { permanent_uuid: "c1136fd063984589a33050dc861d3f52" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40167 } }
I20260928 13:12:23.103645 25587 catalog_manager.cc:5495] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 356d19a9ed554b258f5fd468779161cd (127.8.88.129). New cstate: current_term: 1 leader_uuid: "356d19a9ed554b258f5fd468779161cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "356d19a9ed554b258f5fd468779161cd" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 44495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b51e3e44c56f47a7b4f2417db01129e3" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1136fd063984589a33050dc861d3f52" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40167 } health_report { overall_health: UNKNOWN } } }
W20260928 13:12:23.104156 25779 tablet.cc:2369] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:12:23.117060 26062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.114095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.8.88.129:44495)
0928 13:12:23.114177 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.114180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.114243 (+ 63us) server_negotiation.cc:407] Connection header received
0928 13:12:23.114286 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.114287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.114309 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.114366 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:23.114610 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.115084 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.116065 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.116296 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.116360 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.116543 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:23.116659 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:23.116737 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.116932 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.116963 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 13:12:23.119273 26000 raft_consensus.cc:1261] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52 [term 1 FOLLOWER]: Refusing update from remote peer 356d19a9ed554b258f5fd468779161cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:23.119306 25866 raft_consensus.cc:1261] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3 [term 1 FOLLOWER]: Refusing update from remote peer 356d19a9ed554b258f5fd468779161cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:23.119817 26069 consensus_queue.cc:1035] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b51e3e44c56f47a7b4f2417db01129e3" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:23.120025 26073 consensus_queue.cc:1035] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1136fd063984589a33050dc861d3f52" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:23.123135 26076 mvcc.cc:205] Tried to move back new op lower bound from 7334302282212982784 to 7334302282149507072. Current Snapshot: MvccSnapshot[applied={T|T < 7334302282212982784}]
I20260928 13:12:23.123548 26077 mvcc.cc:205] Tried to move back new op lower bound from 7334302282212982784 to 7334302282149507072. Current Snapshot: MvccSnapshot[applied={T|T < 7334302282212982784}]
I20260928 13:12:23.126101 25586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:12:23.126225 25586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:12:23.127067 25586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d38f9af9e9e84ca9a961db0af6bead89: Sending DeleteTablet for 3 replicas of tablet 5605cef16e3444c2a02685cac2e4712c
I20260928 13:12:23.127545 25846 tablet_service.cc:1514] Processing DeleteTablet for tablet 5605cef16e3444c2a02685cac2e4712c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:23 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:49880
I20260928 13:12:23.127523 25980 tablet_service.cc:1514] Processing DeleteTablet for tablet 5605cef16e3444c2a02685cac2e4712c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:23 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:46658
I20260928 13:12:23.127779 26087 tablet_replica.cc:330] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52: stopping tablet replica
I20260928 13:12:23.127779 25712 tablet_service.cc:1514] Processing DeleteTablet for tablet 5605cef16e3444c2a02685cac2e4712c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:23 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:52138
I20260928 13:12:23.127912 26087 raft_consensus.cc:2229] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:12:23.128047 26087 raft_consensus.cc:2258] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:12:23.128041 26088 tablet_replica.cc:330] T 5605cef16e3444c2a02685cac2e4712c P 356d19a9ed554b258f5fd468779161cd: stopping tablet replica
I20260928 13:12:23.128189 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 25649
I20260928 13:12:23.128521 26087 ts_tablet_manager.cc:1905] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:12:23.129799 26086 tablet_replica.cc:330] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3: stopping tablet replica
I20260928 13:12:23.129947 26086 raft_consensus.cc:2229] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:12:23.130065 26086 raft_consensus.cc:2258] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:12:23.130702 26086 ts_tablet_manager.cc:1905] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:12:23.130728 26087 ts_tablet_manager.cc:1918] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:12:23.130800 26087 log.cc:1196] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/5605cef16e3444c2a02685cac2e4712c
I20260928 13:12:23.131093 26087 ts_tablet_manager.cc:1939] T 5605cef16e3444c2a02685cac2e4712c P c1136fd063984589a33050dc861d3f52: Deleting consensus metadata
I20260928 13:12:23.131740 25572 catalog_manager.cc:4841] TS c1136fd063984589a33050dc861d3f52 (127.8.88.131:40167): tablet 5605cef16e3444c2a02685cac2e4712c (table test-table [id=874b4120c0854447981a6224b5edcd38]) successfully deleted
I20260928 13:12:23.132925 26086 ts_tablet_manager.cc:1918] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:12:23.133039 26086 log.cc:1196] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/5605cef16e3444c2a02685cac2e4712c
I20260928 13:12:23.133356 26086 ts_tablet_manager.cc:1939] T 5605cef16e3444c2a02685cac2e4712c P b51e3e44c56f47a7b4f2417db01129e3: Deleting consensus metadata
I20260928 13:12:23.133769 25574 catalog_manager.cc:4841] TS b51e3e44c56f47a7b4f2417db01129e3 (127.8.88.130:38447): tablet 5605cef16e3444c2a02685cac2e4712c (table test-table [id=874b4120c0854447981a6224b5edcd38]) successfully deleted
W20260928 13:12:23.137436 25573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:44495 (error 108)
I20260928 13:12:23.142072 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.138207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:44495 (local address 127.0.0.1:52160)
0928 13:12:23.141933 (+ 3726us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:23.142018 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:44495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3689}
W20260928 13:12:23.142261 25573 catalog_manager.cc:4584] TS 356d19a9ed554b258f5fd468779161cd (127.8.88.129:44495): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5605cef16e3444c2a02685cac2e4712c: Network error: Client connection negotiation failed: client connection to 127.8.88.129:44495: connect: Connection refused (error 111)
I20260928 13:12:23.142614 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 25783
I20260928 13:12:23.147454 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 25917
I20260928 13:12:23.151926 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 25555
2026-09-28T13:12:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2084 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T13:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:23Z Disabled control of system clock
I20260928 13:12:23.181533 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:44645
--webserver_interface=127.8.88.190
--webserver_port=0
--builtin_ntp_servers=127.8.88.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:44645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:12:23.266839 26093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:23.267089 26093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:23.267112 26093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:23.268399 26093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:23.268450 26093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:23.268463 26093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:23.268476 26093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:23.268515 26093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:23.270104 26093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:44645
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:44645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:23.270398 26093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:23.270743 26093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:12:23.276154 26093 server_base.cc:1034] running on GCE node
W20260928 13:12:23.276233 26099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:23.276347 26101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:23.276170 26098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:23.276831 26093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:23.277215 26093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:23.278409 26093 hybrid_clock.cc:648] HybridClock initialized: now 1790601143278392 us; error 51 us; skew 500 ppm
I20260928 13:12:23.279927 26093 webserver.cc:460] Webserver started at http://127.8.88.190:46833/ using document root <none> and password file <none>
I20260928 13:12:23.280221 26093 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:23.280300 26093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:23.280450 26093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:23.281709 26093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "91bb454655da464cba6f4cbf17c143e7"
format_stamp: "Formatted at 2026-09-28 13:12:23 on dist-test-slave-p1lc"
server_key: "530412fddf43c75367c18fc24804b696"
server_key_iv: "ca04519f0a73c76b99e0efdf43313de1"
server_key_version: "encryptionkey@0"
I20260928 13:12:23.282102 26093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "91bb454655da464cba6f4cbf17c143e7"
format_stamp: "Formatted at 2026-09-28 13:12:23 on dist-test-slave-p1lc"
server_key: "530412fddf43c75367c18fc24804b696"
server_key_iv: "ca04519f0a73c76b99e0efdf43313de1"
server_key_version: "encryptionkey@0"
I20260928 13:12:23.283562 26093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:12:23.284487 26107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:23.284782 26093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:23.284926 26093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "91bb454655da464cba6f4cbf17c143e7"
format_stamp: "Formatted at 2026-09-28 13:12:23 on dist-test-slave-p1lc"
server_key: "530412fddf43c75367c18fc24804b696"
server_key_iv: "ca04519f0a73c76b99e0efdf43313de1"
server_key_version: "encryptionkey@0"
I20260928 13:12:23.285012 26093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:23.293761 26093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:23.294132 26093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:23.294309 26093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:23.300201 26093 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:44645
I20260928 13:12:23.300284 26159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:44645 every 8 connection(s)
I20260928 13:12:23.300668 26093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:23.301777 26160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:23.304277 26160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7: Bootstrap starting.
I20260928 13:12:23.304919 26160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:23.305331 26160 log.cc:824] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:23.306170 26160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7: No bootstrap required, opened a new log
I20260928 13:12:23.306900 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 26093
I20260928 13:12:23.307000 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:23.307181 8546 external_mini_cluster.cc:1428] Setting key 792e38d7f569ed794deba5e8622e9cbc
I20260928 13:12:23.308548 26160 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91bb454655da464cba6f4cbf17c143e7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44645 } }
I20260928 13:12:23.308720 26160 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:23.308759 26160 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91bb454655da464cba6f4cbf17c143e7, State: Initialized, Role: FOLLOWER
I20260928 13:12:23.308974 26160 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91bb454655da464cba6f4cbf17c143e7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44645 } }
I20260928 13:12:23.309095 26160 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:23.309120 26160 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:23.309190 26160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:23.309758 26160 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91bb454655da464cba6f4cbf17c143e7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44645 } }
I20260928 13:12:23.309844 26160 leader_election.cc:302] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91bb454655da464cba6f4cbf17c143e7; no voters:
I20260928 13:12:23.310066 26160 leader_election.cc:288] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:23.310122 26165 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:23.310259 26165 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [term 1 LEADER]: Becoming Leader. State: Replica: 91bb454655da464cba6f4cbf17c143e7, State: Running, Role: LEADER
I20260928 13:12:23.310338 26160 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:23.310395 26165 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91bb454655da464cba6f4cbf17c143e7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44645 } }
I20260928 13:12:23.310762 26165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91bb454655da464cba6f4cbf17c143e7. Latest consensus state: current_term: 1 leader_uuid: "91bb454655da464cba6f4cbf17c143e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91bb454655da464cba6f4cbf17c143e7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44645 } } }
I20260928 13:12:23.310885 26165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:23.310772 26166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91bb454655da464cba6f4cbf17c143e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91bb454655da464cba6f4cbf17c143e7" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 44645 } } }
I20260928 13:12:23.310986 26166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7 [sys.catalog]: This master's current role is: LEADER
W20260928 13:12:23.312352 26181 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:12:23.312410 26181 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:12:23.312855 26171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:23.313118 26171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:23.314261 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.307832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60036 (local address 127.8.88.190:44645)
0928 13:12:23.308078 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.308086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.308112 (+ 26us) server_negotiation.cc:407] Connection header received
0928 13:12:23.308528 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.308534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.308592 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.308707 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:23.308990 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.310663 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.311845 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.312462 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.312592 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.313371 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:23.313375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:23.313375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:23.313590 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:23.313633 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:23.313649 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:23.313709 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.313952 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.314045 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
I20260928 13:12:23.314936 26171 catalog_manager.cc:1347] Generated new cluster ID: 86cf7f492a7b4d95934767c5e658795d
I20260928 13:12:23.314988 26171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:23.320007 26171 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:23.320793 26171 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:23.326670 26171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7: Generated new TSK 0
I20260928 13:12:23.326872 26171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:23.333245 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44645
--builtin_ntp_servers=127.8.88.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:12:23.420390 26184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:23.420636 26184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:23.420661 26184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:23.422027 26184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:23.422080 26184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:23.422109 26184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:12:23.423652 26184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:23.423944 26184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:23.424283 26184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:23.430598 26192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:23.430691 26190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:23.430608 26189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:23.430754 26184 server_base.cc:1034] running on GCE node
I20260928 13:12:23.431192 26184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:23.431517 26184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:23.432711 26184 hybrid_clock.cc:648] HybridClock initialized: now 1790601143432687 us; error 41 us; skew 500 ppm
I20260928 13:12:23.440076 26184 webserver.cc:460] Webserver started at http://127.8.88.129:34327/ using document root <none> and password file <none>
I20260928 13:12:23.440374 26184 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:23.440448 26184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:23.440606 26184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:23.441879 26184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "37e42741ae614a438bb0d7c27f98e714"
format_stamp: "Formatted at 2026-09-28 13:12:23 on dist-test-slave-p1lc"
server_key: "d2f2935f3d097abc7c44887de3d0ce3f"
server_key_iv: "3af2e468c9c90880c948c11c7144d3dd"
server_key_version: "encryptionkey@0"
I20260928 13:12:23.442272 26184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "37e42741ae614a438bb0d7c27f98e714"
format_stamp: "Formatted at 2026-09-28 13:12:23 on dist-test-slave-p1lc"
server_key: "d2f2935f3d097abc7c44887de3d0ce3f"
server_key_iv: "3af2e468c9c90880c948c11c7144d3dd"
server_key_version: "encryptionkey@0"
I20260928 13:12:23.443964 26184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:12:23.445007 26198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:23.445315 26184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:12:23.445433 26184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "37e42741ae614a438bb0d7c27f98e714"
format_stamp: "Formatted at 2026-09-28 13:12:23 on dist-test-slave-p1lc"
server_key: "d2f2935f3d097abc7c44887de3d0ce3f"
server_key_iv: "3af2e468c9c90880c948c11c7144d3dd"
server_key_version: "encryptionkey@0"
I20260928 13:12:23.445531 26184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:23.461299 26184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:23.461689 26184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:23.461859 26184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:23.462124 26184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:23.462488 26184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:23.462607 26184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:23.462651 26184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:23.462680 26184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:23.468753 26184 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:45089
I20260928 13:12:23.468797 26311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:45089 every 8 connection(s)
I20260928 13:12:23.469197 26184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:12:23.474785 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.470080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:40229 (local address 127.8.88.190:44645)
0928 13:12:23.470157 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.470159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.470426 (+ 267us) server_negotiation.cc:407] Connection header received
0928 13:12:23.470931 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.470932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.470944 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.470999 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:23.471731 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.472260 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.473375 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.473613 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.473670 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.474526 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:23.474528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:23.474528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:23.474533 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:23.474549 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:23.474554 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:23.474599 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.474697 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.474721 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 13:12:23.475032 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.470078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:44645 (local address 127.8.88.129:40229)
0928 13:12:23.470301 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:23.470337 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:23.470840 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:23.471054 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:23.471057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:23.471066 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:23.471642 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.471647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.472339 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.472340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.473292 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.473295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.473715 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.473716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.473835 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.474371 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:12:23.474371 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:23.474399 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:12:23.474422 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:23.474624 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:23.474628 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:23.474631 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:12:23.474684 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 13:12:23.474766 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
I20260928 13:12:23.475680 26312 heartbeater.cc:347] Connected to a master server at 127.8.88.190:44645
I20260928 13:12:23.475780 26312 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:23.475998 26312 heartbeater.cc:511] Master 127.8.88.190:44645 requested a full tablet report, sending...
I20260928 13:12:23.476593 26124 ts_manager.cc:194] Registered new tserver with Master: 37e42741ae614a438bb0d7c27f98e714 (127.8.88.129:45089)
I20260928 13:12:23.477227 26124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.88.129:40229
I20260928 13:12:23.478942 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 26184
I20260928 13:12:23.479058 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:12:23.479359 8546 external_mini_cluster.cc:1428] Setting key f8d8b97517235096566ea257c9fae415
I20260928 13:12:23.480453 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44645
--builtin_ntp_servers=127.8.88.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:12:23.565420 26315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:23.565637 26315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:23.565661 26315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:23.566859 26315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:23.566900 26315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:23.566926 26315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:12:23.568315 26315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:23.568571 26315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:23.568894 26315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:23.574208 26323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:23.574193 26321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:23.574190 26320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:23.574504 26315 server_base.cc:1034] running on GCE node
I20260928 13:12:23.574808 26315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:23.575062 26315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:23.576244 26315 hybrid_clock.cc:648] HybridClock initialized: now 1790601143576208 us; error 50 us; skew 500 ppm
I20260928 13:12:23.577906 26315 webserver.cc:460] Webserver started at http://127.8.88.130:43459/ using document root <none> and password file <none>
I20260928 13:12:23.578184 26315 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:23.578262 26315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:23.578428 26315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:23.579607 26315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "b19f84ee9cfa44b6b53e117401445fab"
format_stamp: "Formatted at 2026-09-28 13:12:23 on dist-test-slave-p1lc"
server_key: "06cc41820c00d197d5d929dc46dc2f48"
server_key_iv: "2e9c62421e094d664f8e26686a7486ce"
server_key_version: "encryptionkey@0"
I20260928 13:12:23.579988 26315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "b19f84ee9cfa44b6b53e117401445fab"
format_stamp: "Formatted at 2026-09-28 13:12:23 on dist-test-slave-p1lc"
server_key: "06cc41820c00d197d5d929dc46dc2f48"
server_key_iv: "2e9c62421e094d664f8e26686a7486ce"
server_key_version: "encryptionkey@0"
I20260928 13:12:23.581523 26315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:23.582401 26329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:23.582680 26315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:23.582789 26315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "b19f84ee9cfa44b6b53e117401445fab"
format_stamp: "Formatted at 2026-09-28 13:12:23 on dist-test-slave-p1lc"
server_key: "06cc41820c00d197d5d929dc46dc2f48"
server_key_iv: "2e9c62421e094d664f8e26686a7486ce"
server_key_version: "encryptionkey@0"
I20260928 13:12:23.582878 26315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:23.598935 26315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:23.599273 26315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:23.599431 26315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:23.599684 26315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:23.600013 26315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:23.600067 26315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:23.600112 26315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:23.600145 26315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:23.606032 26315 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:37645
I20260928 13:12:23.606071 26442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:37645 every 8 connection(s)
I20260928 13:12:23.606406 26315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:12:23.612254 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.607295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:41271 (local address 127.8.88.190:44645)
0928 13:12:23.607371 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.607373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.607633 (+ 260us) server_negotiation.cc:407] Connection header received
0928 13:12:23.608179 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.608180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.608193 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.608254 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:23.609106 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.609679 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.610713 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.610962 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.611022 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.612024 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:23.612026 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:23.612026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:23.612031 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:23.612045 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:23.612048 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:23.612090 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.612165 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.612190 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 13:12:23.612649 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.607240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:44645 (local address 127.8.88.130:41271)
0928 13:12:23.607454 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:23.607491 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:23.608063 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:23.608332 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:23.608336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:23.608344 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:23.608851 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.608855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.609786 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.609787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.610620 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.610623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.611057 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.611058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.611174 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.611874 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:12:23.611874 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:23.611905 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:12:23.611929 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:23.612100 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:23.612103 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:23.612105 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 13:12:23.612155 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 13:12:23.612250 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260928 13:12:23.613281 26443 heartbeater.cc:347] Connected to a master server at 127.8.88.190:44645
I20260928 13:12:23.613413 26443 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:23.613668 26443 heartbeater.cc:511] Master 127.8.88.190:44645 requested a full tablet report, sending...
I20260928 13:12:23.614229 26124 ts_manager.cc:194] Registered new tserver with Master: b19f84ee9cfa44b6b53e117401445fab (127.8.88.130:37645)
I20260928 13:12:23.614708 26124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.88.130:41271
I20260928 13:12:23.615810 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 26315
I20260928 13:12:23.615934 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:12:23.616148 8546 external_mini_cluster.cc:1428] Setting key 2ce66ba8262afbbdfff303f66cf60562
I20260928 13:12:23.617151 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44645
--builtin_ntp_servers=127.8.88.148:39795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:12:23.701745 26446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:23.702215 26446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:23.702246 26446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:23.703639 26446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:23.703691 26446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:23.703720 26446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:12:23.705253 26446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:39795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:44645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:23.705521 26446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:23.705842 26446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:23.711369 26451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:23.711614 26446 server_base.cc:1034] running on GCE node
W20260928 13:12:23.711397 26452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:23.711483 26454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:23.712020 26446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:23.712303 26446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:23.713478 26446 hybrid_clock.cc:648] HybridClock initialized: now 1790601143713463 us; error 28 us; skew 500 ppm
I20260928 13:12:23.714808 26446 webserver.cc:460] Webserver started at http://127.8.88.131:38865/ using document root <none> and password file <none>
I20260928 13:12:23.715065 26446 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:23.715137 26446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:23.715293 26446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:23.716457 26446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "7f388bada0214f55a17ee6624da64972"
format_stamp: "Formatted at 2026-09-28 13:12:23 on dist-test-slave-p1lc"
server_key: "49335d7095ffd32a8cad208be1bfdd46"
server_key_iv: "78dc48d6e79589dcadce518929c23e3c"
server_key_version: "encryptionkey@0"
I20260928 13:12:23.716831 26446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "7f388bada0214f55a17ee6624da64972"
format_stamp: "Formatted at 2026-09-28 13:12:23 on dist-test-slave-p1lc"
server_key: "49335d7095ffd32a8cad208be1bfdd46"
server_key_iv: "78dc48d6e79589dcadce518929c23e3c"
server_key_version: "encryptionkey@0"
I20260928 13:12:23.718192 26446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:12:23.719117 26460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:23.719393 26446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:12:23.719520 26446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "7f388bada0214f55a17ee6624da64972"
format_stamp: "Formatted at 2026-09-28 13:12:23 on dist-test-slave-p1lc"
server_key: "49335d7095ffd32a8cad208be1bfdd46"
server_key_iv: "78dc48d6e79589dcadce518929c23e3c"
server_key_version: "encryptionkey@0"
I20260928 13:12:23.719676 26446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:23.730615 26446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:23.730970 26446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:23.731139 26446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:23.731441 26446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:23.731815 26446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:23.731869 26446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:23.731913 26446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:23.731928 26446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:23.737970 26446 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:46839
I20260928 13:12:23.738087 26573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:46839 every 8 connection(s)
I20260928 13:12:23.738413 26446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:12:23.742221 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 26446
I20260928 13:12:23.742331 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:12:23.742528 8546 external_mini_cluster.cc:1428] Setting key 6319775abfd5f900a6870aa1cb95f76c
I20260928 13:12:23.745033 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.739284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:51493 (local address 127.8.88.190:44645)
0928 13:12:23.739363 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.739366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.739652 (+ 286us) server_negotiation.cc:407] Connection header received
0928 13:12:23.740158 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.740159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.740170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.740225 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:23.740982 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.741575 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.743227 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.743549 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.743643 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.744626 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:23.744627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:23.744627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:23.744634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:23.744650 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:23.744654 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:23.744709 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.744856 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.744918 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 13:12:23.745247 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.739267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:44645 (local address 127.8.88.131:51493)
0928 13:12:23.739511 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:23.739556 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:23.740033 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:23.740289 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:23.740293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:23.740302 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:23.740822 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.740827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.741686 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.741687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.743105 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.743108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.743655 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.743656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.743803 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.744342 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:12:23.744343 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:23.744373 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:12:23.744397 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:23.744756 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:23.744763 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:23.744767 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:12:23.744857 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 13:12:23.744985 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":109,"threads_started":1}
I20260928 13:12:23.745738 26574 heartbeater.cc:347] Connected to a master server at 127.8.88.190:44645
I20260928 13:12:23.745828 26574 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:23.746062 26574 heartbeater.cc:511] Master 127.8.88.190:44645 requested a full tablet report, sending...
I20260928 13:12:23.746460 26124 ts_manager.cc:194] Registered new tserver with Master: 7f388bada0214f55a17ee6624da64972 (127.8.88.131:46839)
I20260928 13:12:23.746871 26124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.88.131:51493
I20260928 13:12:23.754895 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:12:23.759999 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.756627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.8.88.190:44645)
0928 13:12:23.756713 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.756715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.756727 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:12:23.756803 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.756804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.756821 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.756936 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:23.757127 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.757703 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.758572 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.758789 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.758873 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.759671 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:23.759673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:23.759673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:23.759680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:23.759698 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:23.759703 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:23.759789 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.759853 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.759885 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 13:12:23.762362 26124 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:12:23.764161 26124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:12:23.771102 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.766747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:37645 (local address 127.0.0.1:35210)
0928 13:12:23.767324 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:23.767338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:23.767401 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:23.767678 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:23.767679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:23.767680 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:23.767715 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.767716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.768950 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.768950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.769942 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.769945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.770854 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.770855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.770916 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.770928 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 13:12:23.770989 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 13:12:23.771034 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":29,"thread_start_us":100,"threads_started":1}
I20260928 13:12:23.771165 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.766870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.8.88.130:37645)
0928 13:12:23.767447 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.767452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.767478 (+ 26us) server_negotiation.cc:407] Connection header received
0928 13:12:23.767517 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.767519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.767534 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.767609 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:23.767789 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.768855 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.770041 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.770762 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.770841 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.770935 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.771023 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.771065 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":83,"threads_started":1}
I20260928 13:12:23.771632 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.766748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:45089 (local address 127.0.0.1:48526)
0928 13:12:23.767042 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:23.767052 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:23.767096 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:23.767739 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:23.767740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:23.767742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:23.767793 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.767795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.768733 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.768734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.770214 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.770217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.771375 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.771376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.771439 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.771442 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:12:23.771508 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 13:12:23.771552 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":92,"thread_start_us":63,"threads_started":1}
I20260928 13:12:23.771708 26587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.767214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.8.88.129:45089)
0928 13:12:23.767405 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.767410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.767420 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:12:23.767461 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.767463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.767491 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.767663 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:23.767871 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.768622 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.770492 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.771265 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.771371 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.771462 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.771573 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.771628 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260928 13:12:23.772207 26377 tablet_service.cc:1467] Processing CreateTablet for tablet c5a49cb3f4934213b07ba7e3c8fe1ee2 (DEFAULT_TABLE table=test-table [id=ae6a3a11240d48af8f38de7479d85db9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:23.772645 26377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a49cb3f4934213b07ba7e3c8fe1ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:23.772785 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.766747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:46839 (local address 127.0.0.1:57274)
0928 13:12:23.766958 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:23.767005 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:23.767086 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:23.767446 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:23.767449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:23.767451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:23.767529 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.767531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.768903 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.768904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.770214 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.770217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.772436 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.772437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.772502 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.772506 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:12:23.772659 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 13:12:23.772710 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":125,"thread_start_us":90,"threads_started":1}
I20260928 13:12:23.772943 26586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.767012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.8.88.131:46839)
0928 13:12:23.767179 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.767184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.767197 (+ 13us) server_negotiation.cc:407] Connection header received
0928 13:12:23.767242 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.767245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.767262 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.767358 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:23.767619 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.768787 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.771515 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.772344 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.772406 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.772504 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.772751 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.772791 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260928 13:12:23.773553 26508 tablet_service.cc:1467] Processing CreateTablet for tablet c5a49cb3f4934213b07ba7e3c8fe1ee2 (DEFAULT_TABLE table=test-table [id=ae6a3a11240d48af8f38de7479d85db9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:23.773880 26508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a49cb3f4934213b07ba7e3c8fe1ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:23.775801 26588 tablet_bootstrap.cc:492] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab: Bootstrap starting.
I20260928 13:12:23.776281 26246 tablet_service.cc:1467] Processing CreateTablet for tablet c5a49cb3f4934213b07ba7e3c8fe1ee2 (DEFAULT_TABLE table=test-table [id=ae6a3a11240d48af8f38de7479d85db9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:23.776544 26588 tablet_bootstrap.cc:654] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:23.776633 26246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a49cb3f4934213b07ba7e3c8fe1ee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:23.776906 26588 log.cc:824] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:23.776943 26589 tablet_bootstrap.cc:492] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972: Bootstrap starting.
I20260928 13:12:23.777609 26589 tablet_bootstrap.cc:654] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:23.777788 26588 tablet_bootstrap.cc:492] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab: No bootstrap required, opened a new log
I20260928 13:12:23.777873 26588 ts_tablet_manager.cc:1397] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:12:23.777944 26589 log.cc:824] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:23.780313 26592 tablet_bootstrap.cc:492] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714: Bootstrap starting.
I20260928 13:12:23.780603 26588 raft_consensus.cc:348] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19f84ee9cfa44b6b53e117401445fab" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37645 } } peers { permanent_uuid: "7f388bada0214f55a17ee6624da64972" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46839 } } peers { permanent_uuid: "37e42741ae614a438bb0d7c27f98e714" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45089 } }
I20260928 13:12:23.780802 26588 raft_consensus.cc:374] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:23.780856 26588 raft_consensus.cc:729] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19f84ee9cfa44b6b53e117401445fab, State: Initialized, Role: FOLLOWER
I20260928 13:12:23.780993 26592 tablet_bootstrap.cc:654] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:23.781030 26588 consensus_queue.cc:260] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19f84ee9cfa44b6b53e117401445fab" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37645 } } peers { permanent_uuid: "7f388bada0214f55a17ee6624da64972" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46839 } } peers { permanent_uuid: "37e42741ae614a438bb0d7c27f98e714" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45089 } }
I20260928 13:12:23.781255 26589 tablet_bootstrap.cc:492] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972: No bootstrap required, opened a new log
I20260928 13:12:23.781319 26588 ts_tablet_manager.cc:1428] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260928 13:12:23.781333 26589 ts_tablet_manager.cc:1397] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 13:12:23.781404 26592 log.cc:824] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:23.781484 26443 heartbeater.cc:503] Master 127.8.88.190:44645 was elected leader, sending a full tablet report...
I20260928 13:12:23.782188 26592 tablet_bootstrap.cc:492] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714: No bootstrap required, opened a new log
I20260928 13:12:23.782275 26592 ts_tablet_manager.cc:1397] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 13:12:23.783840 26589 raft_consensus.cc:348] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19f84ee9cfa44b6b53e117401445fab" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37645 } } peers { permanent_uuid: "7f388bada0214f55a17ee6624da64972" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46839 } } peers { permanent_uuid: "37e42741ae614a438bb0d7c27f98e714" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45089 } }
I20260928 13:12:23.783984 26589 raft_consensus.cc:374] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:23.784006 26589 raft_consensus.cc:729] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f388bada0214f55a17ee6624da64972, State: Initialized, Role: FOLLOWER
I20260928 13:12:23.784159 26589 consensus_queue.cc:260] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19f84ee9cfa44b6b53e117401445fab" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37645 } } peers { permanent_uuid: "7f388bada0214f55a17ee6624da64972" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46839 } } peers { permanent_uuid: "37e42741ae614a438bb0d7c27f98e714" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45089 } }
I20260928 13:12:23.784529 26574 heartbeater.cc:503] Master 127.8.88.190:44645 was elected leader, sending a full tablet report...
I20260928 13:12:23.784528 26589 ts_tablet_manager.cc:1428] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 13:12:23.784654 26592 raft_consensus.cc:348] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19f84ee9cfa44b6b53e117401445fab" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37645 } } peers { permanent_uuid: "7f388bada0214f55a17ee6624da64972" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46839 } } peers { permanent_uuid: "37e42741ae614a438bb0d7c27f98e714" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45089 } }
I20260928 13:12:23.784827 26592 raft_consensus.cc:374] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:23.784902 26592 raft_consensus.cc:729] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e42741ae614a438bb0d7c27f98e714, State: Initialized, Role: FOLLOWER
I20260928 13:12:23.785032 26592 consensus_queue.cc:260] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19f84ee9cfa44b6b53e117401445fab" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37645 } } peers { permanent_uuid: "7f388bada0214f55a17ee6624da64972" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46839 } } peers { permanent_uuid: "37e42741ae614a438bb0d7c27f98e714" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45089 } }
I20260928 13:12:23.785341 26592 ts_tablet_manager.cc:1428] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 13:12:23.785404 26312 heartbeater.cc:503] Master 127.8.88.190:44645 was elected leader, sending a full tablet report...
I20260928 13:12:23.790597 26595 raft_consensus.cc:482] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:12:23.790757 26595 raft_consensus.cc:504] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19f84ee9cfa44b6b53e117401445fab" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37645 } } peers { permanent_uuid: "7f388bada0214f55a17ee6624da64972" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46839 } } peers { permanent_uuid: "37e42741ae614a438bb0d7c27f98e714" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45089 } }
I20260928 13:12:23.791158 26595 leader_election.cc:288] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b19f84ee9cfa44b6b53e117401445fab (127.8.88.130:37645), 37e42741ae614a438bb0d7c27f98e714 (127.8.88.129:45089)
I20260928 13:12:23.794035 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.791333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:45089 (local address 127.8.88.131:41969)
0928 13:12:23.791388 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:23.791401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:23.791442 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:23.791622 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:23.791623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:23.791625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:23.791691 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.791693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.792373 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.792375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.793213 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.793215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.793775 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.793776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.793848 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.793851 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:12:23.793914 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 13:12:23.793960 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 13:12:23.794044 26587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.791405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:41969 (local address 127.8.88.129:45089)
0928 13:12:23.791456 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.791458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.791467 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:12:23.791498 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.791498 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.791511 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.791574 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:23.791771 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.792271 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.793305 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.793689 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.793776 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.793785 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.793932 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.793967 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 13:12:23.794191 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.791359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:37645 (local address 127.8.88.131:38735)
0928 13:12:23.791525 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:23.791536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:23.791600 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:23.791727 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:23.791727 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:23.791729 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:23.791803 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.791805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.792477 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.792479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.793352 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:23.793354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.793918 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.793919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:23.793994 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.793997 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:12:23.794071 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 13:12:23.794110 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":120,"threads_started":1}
I20260928 13:12:23.794184 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.791414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:38735 (local address 127.8.88.130:37645)
0928 13:12:23.791536 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.791539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.791600 (+ 61us) server_negotiation.cc:407] Connection header received
0928 13:12:23.791656 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.791657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.791668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.791724 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:23.791880 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.792387 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.793445 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.793818 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.793915 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.793923 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.794088 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.794126 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 13:12:23.794440 26266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a49cb3f4934213b07ba7e3c8fe1ee2" candidate_uuid: "7f388bada0214f55a17ee6624da64972" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37e42741ae614a438bb0d7c27f98e714" is_pre_election: true
I20260928 13:12:23.794616 26266 raft_consensus.cc:2454] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f388bada0214f55a17ee6624da64972 in term 0.
I20260928 13:12:23.795001 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a49cb3f4934213b07ba7e3c8fe1ee2" candidate_uuid: "7f388bada0214f55a17ee6624da64972" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19f84ee9cfa44b6b53e117401445fab" is_pre_election: true
I20260928 13:12:23.795190 26397 raft_consensus.cc:2454] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f388bada0214f55a17ee6624da64972 in term 0.
I20260928 13:12:23.795421 26462 leader_election.cc:302] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37e42741ae614a438bb0d7c27f98e714, 7f388bada0214f55a17ee6624da64972; no voters:
I20260928 13:12:23.795578 26595 raft_consensus.cc:2781] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:12:23.795644 26595 raft_consensus.cc:482] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:12:23.795661 26595 raft_consensus.cc:3037] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:23.796398 26595 raft_consensus.cc:504] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19f84ee9cfa44b6b53e117401445fab" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37645 } } peers { permanent_uuid: "7f388bada0214f55a17ee6624da64972" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46839 } } peers { permanent_uuid: "37e42741ae614a438bb0d7c27f98e714" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45089 } }
I20260928 13:12:23.796551 26595 leader_election.cc:288] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [CANDIDATE]: Term 1 election: Requested vote from peers b19f84ee9cfa44b6b53e117401445fab (127.8.88.130:37645), 37e42741ae614a438bb0d7c27f98e714 (127.8.88.129:45089)
I20260928 13:12:23.796787 26266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a49cb3f4934213b07ba7e3c8fe1ee2" candidate_uuid: "7f388bada0214f55a17ee6624da64972" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37e42741ae614a438bb0d7c27f98e714"
I20260928 13:12:23.796911 26266 raft_consensus.cc:3037] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:23.797699 26266 raft_consensus.cc:2454] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f388bada0214f55a17ee6624da64972 in term 1.
I20260928 13:12:23.797803 26397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a49cb3f4934213b07ba7e3c8fe1ee2" candidate_uuid: "7f388bada0214f55a17ee6624da64972" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19f84ee9cfa44b6b53e117401445fab"
I20260928 13:12:23.797891 26462 leader_election.cc:302] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37e42741ae614a438bb0d7c27f98e714, 7f388bada0214f55a17ee6624da64972; no voters:
I20260928 13:12:23.797917 26397 raft_consensus.cc:3037] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:23.798039 26595 raft_consensus.cc:2781] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:23.798236 26595 raft_consensus.cc:686] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [term 1 LEADER]: Becoming Leader. State: Replica: 7f388bada0214f55a17ee6624da64972, State: Running, Role: LEADER
I20260928 13:12:23.798342 26595 consensus_queue.cc:237] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19f84ee9cfa44b6b53e117401445fab" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37645 } } peers { permanent_uuid: "7f388bada0214f55a17ee6624da64972" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46839 } } peers { permanent_uuid: "37e42741ae614a438bb0d7c27f98e714" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45089 } }
I20260928 13:12:23.798893 26397 raft_consensus.cc:2454] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f388bada0214f55a17ee6624da64972 in term 1.
I20260928 13:12:23.799228 26124 catalog_manager.cc:5495] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f388bada0214f55a17ee6624da64972 (127.8.88.131). New cstate: current_term: 1 leader_uuid: "7f388bada0214f55a17ee6624da64972" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19f84ee9cfa44b6b53e117401445fab" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f388bada0214f55a17ee6624da64972" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37e42741ae614a438bb0d7c27f98e714" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45089 } health_report { overall_health: UNKNOWN } } }
I20260928 13:12:23.808152 26586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.805388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.8.88.131:46839)
0928 13:12:23.805458 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:23.805460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:23.805489 (+ 29us) server_negotiation.cc:407] Connection header received
0928 13:12:23.805526 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:23.805527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:23.805544 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:23.805606 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:23.805709 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.806243 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.807187 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:23.807389 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:23.807459 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:23.807695 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:23.807801 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:23.807867 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:23.807973 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 13:12:23.807996 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 13:12:23.810135 26397 raft_consensus.cc:1261] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab [term 1 FOLLOWER]: Refusing update from remote peer 7f388bada0214f55a17ee6624da64972: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:23.810189 26266 raft_consensus.cc:1261] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714 [term 1 FOLLOWER]: Refusing update from remote peer 7f388bada0214f55a17ee6624da64972: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:23.810570 26595 consensus_queue.cc:1035] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19f84ee9cfa44b6b53e117401445fab" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:23.810822 26598 consensus_queue.cc:1035] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37e42741ae614a438bb0d7c27f98e714" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 45089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:23.813731 26601 mvcc.cc:205] Tried to move back new op lower bound from 7334302285043654656 to 7334302284998750208. Current Snapshot: MvccSnapshot[applied={T|T < 7334302285043654656}]
I20260928 13:12:23.814073 26602 mvcc.cc:205] Tried to move back new op lower bound from 7334302285043654656 to 7334302284998750208. Current Snapshot: MvccSnapshot[applied={T|T < 7334302285043654656}]
I20260928 13:12:23.814834 26604 mvcc.cc:205] Tried to move back new op lower bound from 7334302285043654656 to 7334302284998750208. Current Snapshot: MvccSnapshot[applied={T|T < 7334302285043654656 or (T in {7334302285043654656})}]
I20260928 13:12:23.816709 26123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:12:23.816817 26123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:12:23.817716 26123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91bb454655da464cba6f4cbf17c143e7: Sending DeleteTablet for 3 replicas of tablet c5a49cb3f4934213b07ba7e3c8fe1ee2
I20260928 13:12:23.818073 26377 tablet_service.cc:1514] Processing DeleteTablet for tablet c5a49cb3f4934213b07ba7e3c8fe1ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:23 UTC) from {username='slave'} at 127.0.0.1:35210
I20260928 13:12:23.818092 26246 tablet_service.cc:1514] Processing DeleteTablet for tablet c5a49cb3f4934213b07ba7e3c8fe1ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:23 UTC) from {username='slave'} at 127.0.0.1:48526
I20260928 13:12:23.818349 26613 tablet_replica.cc:330] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 37e42741ae614a438bb0d7c27f98e714: stopping tablet replica
I20260928 13:12:23.818480 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 26184
I20260928 13:12:23.819453 26612 tablet_replica.cc:330] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab: stopping tablet replica
I20260928 13:12:23.819573 26612 raft_consensus.cc:2229] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:12:23.819705 26612 raft_consensus.cc:2258] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:12:23.819984 26508 tablet_service.cc:1514] Processing DeleteTablet for tablet c5a49cb3f4934213b07ba7e3c8fe1ee2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:23 UTC) from {username='slave'} at 127.0.0.1:57274
I20260928 13:12:23.820214 26614 tablet_replica.cc:330] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972: stopping tablet replica
I20260928 13:12:23.820290 26614 raft_consensus.cc:2229] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:12:23.820364 26614 raft_consensus.cc:2258] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:12:23.820771 26614 ts_tablet_manager.cc:1905] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:12:23.823436 26614 ts_tablet_manager.cc:1918] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:12:23.823525 26614 log.cc:1196] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/c5a49cb3f4934213b07ba7e3c8fe1ee2
I20260928 13:12:23.823808 26614 ts_tablet_manager.cc:1939] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P 7f388bada0214f55a17ee6624da64972: Deleting consensus metadata
I20260928 13:12:23.824579 26108 catalog_manager.cc:4841] TS 7f388bada0214f55a17ee6624da64972 (127.8.88.131:46839): tablet c5a49cb3f4934213b07ba7e3c8fe1ee2 (table test-table [id=ae6a3a11240d48af8f38de7479d85db9]) successfully deleted
I20260928 13:12:23.825476 26612 ts_tablet_manager.cc:1905] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 13:12:23.826706 26109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:45089 (error 108)
I20260928 13:12:23.827250 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.827115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:45089 (local address 127.0.0.1:48534)
0928 13:12:23.827173 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:23.827213 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:45089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 13:12:23.827374 26109 catalog_manager.cc:4584] TS 37e42741ae614a438bb0d7c27f98e714 (127.8.88.129:45089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5a49cb3f4934213b07ba7e3c8fe1ee2: Network error: Client connection negotiation failed: client connection to 127.8.88.129:45089: connect: Connection refused (error 111)
I20260928 13:12:23.827667 26612 ts_tablet_manager.cc:1918] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:12:23.827756 26612 log.cc:1196] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/c5a49cb3f4934213b07ba7e3c8fe1ee2
I20260928 13:12:23.827931 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 26315
I20260928 13:12:23.828059 26612 ts_tablet_manager.cc:1939] T c5a49cb3f4934213b07ba7e3c8fe1ee2 P b19f84ee9cfa44b6b53e117401445fab: Deleting consensus metadata
I20260928 13:12:23.833473 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 26446
I20260928 13:12:23.833500 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:23.833359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:37645 (local address 127.0.0.1:35224)
0928 13:12:23.833419 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:23.833456 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.130:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 13:12:23.838655 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 26093
2026-09-28T13:12:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (685 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:23 dist-test-slave-p1lc krb5kdc[26618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:23 dist-test-slave-p1lc krb5kdc[26618](info): set up 2 sockets
Sep 28 13:12:23 dist-test-slave-p1lc krb5kdc[26618](info): commencing operation
krb5kdc: starting...
W20260928 13:12:24.905059 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:24 dist-test-slave-p1lc krb5kdc[26618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:12:25.035961 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:45293
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:45293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:25.121605 26631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:25.121872 26631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:25.121914 26631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:25.123165 26631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:25.123257 26631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:25.123282 26631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:25.123306 26631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:25.123335 26631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:25.124993 26631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:45293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:45293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:25.125341 26631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:25.125617 26631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:12:25.131191 26631 server_base.cc:1034] running on GCE node
W20260928 13:12:25.131192 26637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:25.131155 26636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:25.131147 26639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:25.131927 26631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:25.132272 26631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:25.133453 26631 hybrid_clock.cc:648] HybridClock initialized: now 1790601145133443 us; error 57 us; skew 500 ppm
Sep 28 13:12:25 dist-test-slave-p1lc krb5kdc[26618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601145, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:25.136181 26631 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:12:25.136727 26631 webserver.cc:460] Webserver started at http://127.8.88.190:43163/ using document root <none> and password file <none>
I20260928 13:12:25.137035 26631 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:25.137107 26631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:25.137282 26631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:25.138475 26631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "6f3790b9d2534cb2863d0cce247eea78"
format_stamp: "Formatted at 2026-09-28 13:12:25 on dist-test-slave-p1lc"
server_key: "3fd8d194003000350366fbf08cda77ff"
server_key_iv: "d9d89390a8a10350d29c0ed8456b52a4"
server_key_version: "encryptionkey@0"
I20260928 13:12:25.138862 26631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "6f3790b9d2534cb2863d0cce247eea78"
format_stamp: "Formatted at 2026-09-28 13:12:25 on dist-test-slave-p1lc"
server_key: "3fd8d194003000350366fbf08cda77ff"
server_key_iv: "d9d89390a8a10350d29c0ed8456b52a4"
server_key_version: "encryptionkey@0"
I20260928 13:12:25.140583 26631 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 13:12:25.141644 26646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:25.141880 26631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:25.141989 26631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "6f3790b9d2534cb2863d0cce247eea78"
format_stamp: "Formatted at 2026-09-28 13:12:25 on dist-test-slave-p1lc"
server_key: "3fd8d194003000350366fbf08cda77ff"
server_key_iv: "d9d89390a8a10350d29c0ed8456b52a4"
server_key_version: "encryptionkey@0"
I20260928 13:12:25.142089 26631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:25.158414 26631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:25.160771 26631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:25.161015 26631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:25.165510 26631 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:45293
I20260928 13:12:25.165591 26698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:45293 every 8 connection(s)
I20260928 13:12:25.165966 26631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:25.166868 26699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:25.169286 26699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78: Bootstrap starting.
I20260928 13:12:25.169818 26699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:25.170055 26699 log.cc:824] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:25.170743 26699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78: No bootstrap required, opened a new log
I20260928 13:12:25.172303 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 26631
I20260928 13:12:25.172433 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:25.172472 26699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3790b9d2534cb2863d0cce247eea78" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 45293 } }
I20260928 13:12:25.172626 8546 external_mini_cluster.cc:1428] Setting key 15f2fbbe2a1a2a1f294cd1daa6f05dd5
I20260928 13:12:25.172662 26699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:25.172703 26699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3790b9d2534cb2863d0cce247eea78, State: Initialized, Role: FOLLOWER
I20260928 13:12:25.172905 26699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3790b9d2534cb2863d0cce247eea78" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 45293 } }
I20260928 13:12:25.173010 26699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:25.173058 26699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:25.173108 26699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:25.174185 26699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3790b9d2534cb2863d0cce247eea78" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 45293 } }
I20260928 13:12:25.174309 26699 leader_election.cc:302] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f3790b9d2534cb2863d0cce247eea78; no voters:
I20260928 13:12:25.174480 26699 leader_election.cc:288] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:25.174510 26704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:25.174664 26704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [term 1 LEADER]: Becoming Leader. State: Replica: 6f3790b9d2534cb2863d0cce247eea78, State: Running, Role: LEADER
I20260928 13:12:25.174752 26699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:25.174765 26704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3790b9d2534cb2863d0cce247eea78" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 45293 } }
I20260928 13:12:25.177146 26705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f3790b9d2534cb2863d0cce247eea78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3790b9d2534cb2863d0cce247eea78" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 45293 } } }
I20260928 13:12:25.177199 26705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:25.177021 26706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f3790b9d2534cb2863d0cce247eea78. Latest consensus state: current_term: 1 leader_uuid: "6f3790b9d2534cb2863d0cce247eea78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f3790b9d2534cb2863d0cce247eea78" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 45293 } } }
I20260928 13:12:25.177405 26706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78 [sys.catalog]: This master's current role is: LEADER
W20260928 13:12:25.177461 26718 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:12:25.177552 26718 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:12:25.177757 26720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:25.177927 26720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:25.179394 26720 catalog_manager.cc:1347] Generated new cluster ID: 8fbc86cc972546d780fe71e1bc9bc4c7
I20260928 13:12:25.179447 26720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:12:25 dist-test-slave-p1lc krb5kdc[26618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:25.184688 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.173236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.8.88.190:45293)
0928 13:12:25.173462 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:25.173470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:25.173497 (+ 27us) server_negotiation.cc:407] Connection header received
0928 13:12:25.173620 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:25.173627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:25.173676 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:25.173807 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:25.174699 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.176300 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.177443 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.178065 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.178156 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.181123 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:25.181127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:25.181127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:25.181140 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:25.183050 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:25.183431 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:25.183434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:25.183435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:25.183538 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:25.183775 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:25.183776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:25.183776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:25.184023 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:25.184199 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:25.184309 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 13:12:25.184460 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":128,"threads_started":1}
I20260928 13:12:25.205821 26720 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:25.206434 26720 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:25.212072 26720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78: Generated new TSK 0
I20260928 13:12:25.212376 26720 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:12:25.244227 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:45293
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:25.330775 26725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:25.331033 26725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:25.331075 26725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:25.332346 26725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:25.332419 26725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:25.332466 26725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:12:25.334091 26725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:45293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:25.334378 26725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:25.334640 26725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:25.340236 26733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:25.340294 26731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:25.340364 26725 server_base.cc:1034] running on GCE node
W20260928 13:12:25.340514 26730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:25.340845 26725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:25.341197 26725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:25.342360 26725 hybrid_clock.cc:648] HybridClock initialized: now 1790601145342347 us; error 31 us; skew 500 ppm
Sep 28 13:12:25 dist-test-slave-p1lc krb5kdc[26618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601145, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:25.345372 26725 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:12:25.345911 26725 webserver.cc:460] Webserver started at http://127.8.88.129:37459/ using document root <none> and password file <none>
I20260928 13:12:25.346113 26725 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:25.346156 26725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:25.346253 26725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:25.347307 26725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "0c18584d04d148a6896f90826acdcfbb"
format_stamp: "Formatted at 2026-09-28 13:12:25 on dist-test-slave-p1lc"
server_key: "b9e208b91f639ebb777434de37003ad8"
server_key_iv: "74adeac81b9a3ed4e62b32c840dbd6be"
server_key_version: "encryptionkey@0"
I20260928 13:12:25.347643 26725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "0c18584d04d148a6896f90826acdcfbb"
format_stamp: "Formatted at 2026-09-28 13:12:25 on dist-test-slave-p1lc"
server_key: "b9e208b91f639ebb777434de37003ad8"
server_key_iv: "74adeac81b9a3ed4e62b32c840dbd6be"
server_key_version: "encryptionkey@0"
I20260928 13:12:25.349077 26725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:25.350106 26740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:25.350370 26725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:25.350446 26725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "0c18584d04d148a6896f90826acdcfbb"
format_stamp: "Formatted at 2026-09-28 13:12:25 on dist-test-slave-p1lc"
server_key: "b9e208b91f639ebb777434de37003ad8"
server_key_iv: "74adeac81b9a3ed4e62b32c840dbd6be"
server_key_version: "encryptionkey@0"
I20260928 13:12:25.350546 26725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:25.369912 26725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:25.372440 26725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:25.372568 26725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:25.372915 26725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:25.373251 26725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:25.373287 26725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:25.373311 26725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:25.373356 26725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:25.379848 26725 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:39403
I20260928 13:12:25.379946 26853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:39403 every 8 connection(s)
I20260928 13:12:25.380285 26725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
Sep 28 13:12:25 dist-test-slave-p1lc krb5kdc[26618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601145, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:25.390395 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 26725
I20260928 13:12:25.390538 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:12:25.390754 8546 external_mini_cluster.cc:1428] Setting key 93c822933549b4915d5e1ef41d2a10f2
I20260928 13:12:25.392321 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.381227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:34931 (local address 127.8.88.190:45293)
0928 13:12:25.381370 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:25.381374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:25.381627 (+ 253us) server_negotiation.cc:407] Connection header received
0928 13:12:25.382243 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:25.382245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:25.382258 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:25.382317 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:25.383442 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.384027 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.385262 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.385493 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.385584 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.388308 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:25.388310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:25.388310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:25.388318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:25.390219 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:25.390874 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:25.390875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:25.390876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:25.390954 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:25.391452 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:25.391453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:25.391453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:25.391642 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:25.391708 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:25.392057 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 13:12:25.392162 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 13:12:25.392349 26856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.381187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:45293 (local address 127.8.88.129:34931)
0928 13:12:25.381417 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:25.381469 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:25.382117 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:25.382419 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:25.382422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:25.383077 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:25.383327 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.383332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.384150 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.384151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.385158 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.385161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.385597 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.385599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.385739 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.386392 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:25.386402 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:25.388183 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:25.390373 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:25.390384 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:25.390387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:25.390764 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:25.391031 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:25.391032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:25.391032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:25.391224 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:25.391724 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:25.391732 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:25.391905 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 13:12:25.392004 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 13:12:25.392164 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":93,"threads_started":1}
I20260928 13:12:25.392940 26854 heartbeater.cc:347] Connected to a master server at 127.8.88.190:45293
I20260928 13:12:25.393046 26854 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:25.393262 26854 heartbeater.cc:511] Master 127.8.88.190:45293 requested a full tablet report, sending...
I20260928 13:12:25.393905 26663 ts_manager.cc:194] Registered new tserver with Master: 0c18584d04d148a6896f90826acdcfbb (127.8.88.129:39403)
I20260928 13:12:25.394541 26663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:34931
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:12:25.422029 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:45293
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:25.509660 26859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:25.509932 26859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:25.509974 26859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:25.511221 26859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:25.511299 26859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:25.511348 26859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:12:25.512953 26859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:45293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:25.513284 26859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:25.513545 26859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:12:25.519523 26859 server_base.cc:1034] running on GCE node
W20260928 13:12:25.519503 26867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:25.519503 26864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:25.519500 26865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:25.520016 26859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:25.520262 26859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:25.521418 26859 hybrid_clock.cc:648] HybridClock initialized: now 1790601145521405 us; error 30 us; skew 500 ppm
Sep 28 13:12:25 dist-test-slave-p1lc krb5kdc[26618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601145, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:25.524037 26859 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:12:25.524515 26859 webserver.cc:460] Webserver started at http://127.8.88.130:46639/ using document root <none> and password file <none>
I20260928 13:12:25.524775 26859 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:25.524847 26859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:25.525048 26859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:25.526187 26859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "3c6479b7b3cc4293b640e8be81143361"
format_stamp: "Formatted at 2026-09-28 13:12:25 on dist-test-slave-p1lc"
server_key: "40e9b9569ff2712ae3c41de0e586f210"
server_key_iv: "af9e4312b087099fb9975e2d6c1880d1"
server_key_version: "encryptionkey@0"
I20260928 13:12:25.526561 26859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "3c6479b7b3cc4293b640e8be81143361"
format_stamp: "Formatted at 2026-09-28 13:12:25 on dist-test-slave-p1lc"
server_key: "40e9b9569ff2712ae3c41de0e586f210"
server_key_iv: "af9e4312b087099fb9975e2d6c1880d1"
server_key_version: "encryptionkey@0"
I20260928 13:12:25.527961 26859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:25.528926 26874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:25.529145 26859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:25.529250 26859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "3c6479b7b3cc4293b640e8be81143361"
format_stamp: "Formatted at 2026-09-28 13:12:25 on dist-test-slave-p1lc"
server_key: "40e9b9569ff2712ae3c41de0e586f210"
server_key_iv: "af9e4312b087099fb9975e2d6c1880d1"
server_key_version: "encryptionkey@0"
I20260928 13:12:25.529340 26859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:25.538776 26859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:25.541066 26859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:25.541260 26859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:25.541539 26859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:25.541944 26859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:25.542006 26859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:25.542050 26859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:25.542083 26859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:25.548532 26859 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:37847
I20260928 13:12:25.548610 26987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:37847 every 8 connection(s)
I20260928 13:12:25.549021 26859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
Sep 28 13:12:25 dist-test-slave-p1lc krb5kdc[26618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601145, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:25.557993 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 26859
I20260928 13:12:25.558089 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:12:25.558283 8546 external_mini_cluster.cc:1428] Setting key 6ac3937cb5d85b00c9ee37cacfacd83a
I20260928 13:12:25.559965 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.550010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:38673 (local address 127.8.88.190:45293)
0928 13:12:25.550107 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:25.550109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:25.550229 (+ 120us) server_negotiation.cc:407] Connection header received
0928 13:12:25.550781 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:25.550782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:25.550792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:25.550848 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:25.551928 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.552444 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.553807 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.554016 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.554076 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.556514 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:25.556515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:25.556515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:25.556523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:25.558039 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:25.558677 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:25.558678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:25.558678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:25.558742 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:25.559031 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:25.559031 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:25.559032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:25.559257 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:25.559345 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:25.559719 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 13:12:25.559820 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 13:12:25.560014 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.549920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:45293 (local address 127.8.88.130:38673)
0928 13:12:25.550082 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:25.550142 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:25.550664 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:25.550908 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:25.550912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:25.551592 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:25.551829 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.551835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.552541 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.552542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.553645 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.553648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.554146 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.554148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.554276 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.554932 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:25.554943 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:25.556413 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:25.558210 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:25.558217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:25.558220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:25.558588 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:25.558821 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:25.558822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:25.558823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:25.558962 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:25.559368 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:25.559375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:25.559533 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 13:12:25.559661 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 13:12:25.559811 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260928 13:12:25.560600 26988 heartbeater.cc:347] Connected to a master server at 127.8.88.190:45293
I20260928 13:12:25.560717 26988 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:25.560981 26988 heartbeater.cc:511] Master 127.8.88.190:45293 requested a full tablet report, sending...
I20260928 13:12:25.561467 26663 ts_manager.cc:194] Registered new tserver with Master: 3c6479b7b3cc4293b640e8be81143361 (127.8.88.130:37847)
I20260928 13:12:25.561841 26663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:38673
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:12:25.586658 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:45293
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:25.671396 26993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:25.671670 26993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:25.671715 26993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:25.673028 26993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:25.673115 26993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:25.673166 26993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:12:25.674858 26993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:45293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:25.675168 26993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:25.675616 26993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:25.681237 27001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:25.681403 26998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:25.681237 26999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:25.681492 26993 server_base.cc:1034] running on GCE node
I20260928 13:12:25.681886 26993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:25.682209 26993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:25.683418 26993 hybrid_clock.cc:648] HybridClock initialized: now 1790601145683410 us; error 43 us; skew 500 ppm
Sep 28 13:12:25 dist-test-slave-p1lc krb5kdc[26618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601145, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:25.686276 26993 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:12:25.686782 26993 webserver.cc:460] Webserver started at http://127.8.88.131:43737/ using document root <none> and password file <none>
I20260928 13:12:25.686978 26993 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:25.687021 26993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:25.687121 26993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:25.688200 26993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "e1cc39c92baa460588c4a853779da34b"
format_stamp: "Formatted at 2026-09-28 13:12:25 on dist-test-slave-p1lc"
server_key: "c6801de2914374230c3c64dfb773c0d0"
server_key_iv: "e35fdbdb7505a1b6814897bb1eb6566d"
server_key_version: "encryptionkey@0"
I20260928 13:12:25.688545 26993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "e1cc39c92baa460588c4a853779da34b"
format_stamp: "Formatted at 2026-09-28 13:12:25 on dist-test-slave-p1lc"
server_key: "c6801de2914374230c3c64dfb773c0d0"
server_key_iv: "e35fdbdb7505a1b6814897bb1eb6566d"
server_key_version: "encryptionkey@0"
I20260928 13:12:25.690002 26993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:12:25.690814 27008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:25.691075 26993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:12:25.691198 26993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "e1cc39c92baa460588c4a853779da34b"
format_stamp: "Formatted at 2026-09-28 13:12:25 on dist-test-slave-p1lc"
server_key: "c6801de2914374230c3c64dfb773c0d0"
server_key_iv: "e35fdbdb7505a1b6814897bb1eb6566d"
server_key_version: "encryptionkey@0"
I20260928 13:12:25.691291 26993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:25.711208 26993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:25.713939 26993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:25.714126 26993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:25.714365 26993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:25.714696 26993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:25.714751 26993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:25.714792 26993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:25.714818 26993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:25.721112 26993 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:40181
I20260928 13:12:25.721163 27121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:40181 every 8 connection(s)
I20260928 13:12:25.721555 26993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:12:25.722473 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 26993
I20260928 13:12:25.722584 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:12:25.722765 8546 external_mini_cluster.cc:1428] Setting key ecaa37c8bb695e0926164ef59d59eafa
Sep 28 13:12:25 dist-test-slave-p1lc krb5kdc[26618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601145, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:25.734036 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.722374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:54741 (local address 127.8.88.190:45293)
0928 13:12:25.722424 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:25.722435 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:25.722665 (+ 230us) server_negotiation.cc:407] Connection header received
0928 13:12:25.723364 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:25.723365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:25.723379 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:25.723433 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:25.724454 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.725023 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.728056 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.728278 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.728370 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.730827 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:25.730828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:25.730828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:25.730836 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:25.732435 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:25.732987 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:25.732989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:25.732989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:25.733054 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:25.733350 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:25.733350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:25.733351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:25.733476 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:25.733555 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:25.733858 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 13:12:25.733942 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:12:25.734275 27124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.722270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:45293 (local address 127.8.88.131:54741)
0928 13:12:25.722508 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:25.722575 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:25.723260 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:25.723502 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:25.723507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:25.724161 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:25.724356 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.724361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.725111 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.725112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.727923 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.727928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.728384 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.728385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.728530 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.729174 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:25.729183 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:25.730702 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:25.732556 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:25.732564 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:25.732566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:25.732862 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:25.733126 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:25.733127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:25.733128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:25.733286 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:25.733546 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:25.733553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:25.733708 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 13:12:25.733821 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 13:12:25.733996 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
I20260928 13:12:25.734834 27122 heartbeater.cc:347] Connected to a master server at 127.8.88.190:45293
I20260928 13:12:25.734956 27122 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:25.735141 27122 heartbeater.cc:511] Master 127.8.88.190:45293 requested a full tablet report, sending...
I20260928 13:12:25.735750 26663 ts_manager.cc:194] Registered new tserver with Master: e1cc39c92baa460588c4a853779da34b (127.8.88.131:40181)
I20260928 13:12:25.736174 26663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:54741
I20260928 13:12:25.746541 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:12:25.755125 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.748058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.8.88.190:45293)
0928 13:12:25.748147 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:25.748150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:25.748240 (+ 90us) server_negotiation.cc:407] Connection header received
0928 13:12:25.748334 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:25.748335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:25.748351 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:25.748415 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:25.749057 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.749561 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.750346 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.750541 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.750654 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.752098 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:25.752099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:25.752100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:25.752108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:25.753757 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:25.754209 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:25.754210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:25.754210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:25.754279 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:25.754513 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:25.754514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:25.754514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:25.754664 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:25.754788 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:25.754947 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 13:12:25.755015 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 13:12:25.757267 26663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:12:25.759130 26663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:12:25.766767 27133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.761970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:40181 (local address 127.0.0.1:35084)
0928 13:12:25.762216 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:25.762260 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:25.762347 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:25.762679 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:25.762682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:25.762684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:25.762764 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.762765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.764238 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.764239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.765451 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.765455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.766358 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.766359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.766463 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.766481 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 13:12:25.766552 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 13:12:25.766637 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":49,"thread_start_us":93,"threads_started":1}
I20260928 13:12:25.766835 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.762088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.8.88.130:37847)
0928 13:12:25.762724 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:25.762770 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:25.762780 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:12:25.762930 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:25.762933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:25.762950 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:25.763037 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:25.763237 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.764285 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.765840 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.766394 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.766463 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.766520 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:25.766664 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 13:12:25.766745 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":90,"threads_started":1}
I20260928 13:12:25.766778 27134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.761977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:37847 (local address 127.0.0.1:48860)
0928 13:12:25.762321 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:25.762331 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:25.762373 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:25.763075 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:25.763076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:25.763079 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:25.763138 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.763140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.764442 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.764443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.765738 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.765740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.766481 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.766482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.766558 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.766562 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:12:25.766638 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 13:12:25.766686 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":27,"thread_start_us":88,"threads_started":1}
I20260928 13:12:25.767248 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.762067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35084 (local address 127.8.88.131:40181)
0928 13:12:25.762268 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:25.762273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:25.762437 (+ 164us) server_negotiation.cc:407] Connection header received
0928 13:12:25.762485 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:25.762488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:25.762509 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:25.762578 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:25.762859 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.764111 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.765558 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.766253 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.766339 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.766404 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:25.767110 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 13:12:25.767165 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260928 13:12:25.767462 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.762220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:39403 (local address 127.0.0.1:56962)
0928 13:12:25.762337 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:25.762349 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:25.762547 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:25.763368 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:25.763369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:25.763371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:25.763407 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.763408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.764374 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.764375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.765737 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.765740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.766740 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.766740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.766800 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.766804 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:12:25.767342 (+ 538us) client_negotiation.cc:240] Negotiation successful
0928 13:12:25.767396 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":79,"thread_start_us":73,"threads_started":1}
I20260928 13:12:25.767812 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.762448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56962 (local address 127.8.88.129:39403)
0928 13:12:25.763096 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:25.763100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:25.763112 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:12:25.763153 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:25.763155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:25.763171 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:25.763280 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:25.763488 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.764261 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.765838 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.766590 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.766655 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.766705 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:25.767612 (+ 907us) server_negotiation.cc:299] Negotiation successful
0928 13:12:25.767692 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":80,"threads_started":1}
I20260928 13:12:25.767861 26922 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2eb1b5b4d0481aa7be54bfebb83371 (DEFAULT_TABLE table=test-table [id=62a4a309a1f340998e52d95c52def925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:25.768035 27056 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2eb1b5b4d0481aa7be54bfebb83371 (DEFAULT_TABLE table=test-table [id=62a4a309a1f340998e52d95c52def925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:25.768272 26922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2eb1b5b4d0481aa7be54bfebb83371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:25.768425 27056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2eb1b5b4d0481aa7be54bfebb83371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:25.768505 26788 tablet_service.cc:1467] Processing CreateTablet for tablet 7d2eb1b5b4d0481aa7be54bfebb83371 (DEFAULT_TABLE table=test-table [id=62a4a309a1f340998e52d95c52def925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:25.768816 26788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d2eb1b5b4d0481aa7be54bfebb83371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:25.771792 27139 tablet_bootstrap.cc:492] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b: Bootstrap starting.
I20260928 13:12:25.772301 27140 tablet_bootstrap.cc:492] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361: Bootstrap starting.
I20260928 13:12:25.772518 27141 tablet_bootstrap.cc:492] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb: Bootstrap starting.
I20260928 13:12:25.773110 27140 tablet_bootstrap.cc:654] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:25.773123 27139 tablet_bootstrap.cc:654] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:25.773223 27141 tablet_bootstrap.cc:654] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:25.773690 27141 log.cc:824] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:25.773759 27140 log.cc:824] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:25.773795 27139 log.cc:824] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:25.774633 27141 tablet_bootstrap.cc:492] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb: No bootstrap required, opened a new log
I20260928 13:12:25.774633 27140 tablet_bootstrap.cc:492] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361: No bootstrap required, opened a new log
I20260928 13:12:25.774792 27139 tablet_bootstrap.cc:492] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b: No bootstrap required, opened a new log
I20260928 13:12:25.774796 27141 ts_tablet_manager.cc:1397] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 13:12:25.774796 27140 ts_tablet_manager.cc:1397] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 13:12:25.774868 27139 ts_tablet_manager.cc:1397] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 13:12:25.776933 27139 raft_consensus.cc:348] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cc39c92baa460588c4a853779da34b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40181 } } peers { permanent_uuid: "3c6479b7b3cc4293b640e8be81143361" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37847 } } peers { permanent_uuid: "0c18584d04d148a6896f90826acdcfbb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39403 } }
I20260928 13:12:25.777129 27139 raft_consensus.cc:374] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:25.777176 27139 raft_consensus.cc:729] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cc39c92baa460588c4a853779da34b, State: Initialized, Role: FOLLOWER
I20260928 13:12:25.777302 27139 consensus_queue.cc:260] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cc39c92baa460588c4a853779da34b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40181 } } peers { permanent_uuid: "3c6479b7b3cc4293b640e8be81143361" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37847 } } peers { permanent_uuid: "0c18584d04d148a6896f90826acdcfbb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39403 } }
I20260928 13:12:25.777074 27140 raft_consensus.cc:348] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cc39c92baa460588c4a853779da34b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40181 } } peers { permanent_uuid: "3c6479b7b3cc4293b640e8be81143361" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37847 } } peers { permanent_uuid: "0c18584d04d148a6896f90826acdcfbb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39403 } }
I20260928 13:12:25.777148 27141 raft_consensus.cc:348] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cc39c92baa460588c4a853779da34b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40181 } } peers { permanent_uuid: "3c6479b7b3cc4293b640e8be81143361" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37847 } } peers { permanent_uuid: "0c18584d04d148a6896f90826acdcfbb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39403 } }
I20260928 13:12:25.777431 27140 raft_consensus.cc:374] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:25.777485 27140 raft_consensus.cc:729] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c6479b7b3cc4293b640e8be81143361, State: Initialized, Role: FOLLOWER
I20260928 13:12:25.777488 27141 raft_consensus.cc:374] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:25.777626 27141 raft_consensus.cc:729] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c18584d04d148a6896f90826acdcfbb, State: Initialized, Role: FOLLOWER
I20260928 13:12:25.777639 27139 ts_tablet_manager.cc:1428] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:12:25.777637 27140 consensus_queue.cc:260] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cc39c92baa460588c4a853779da34b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40181 } } peers { permanent_uuid: "3c6479b7b3cc4293b640e8be81143361" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37847 } } peers { permanent_uuid: "0c18584d04d148a6896f90826acdcfbb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39403 } }
I20260928 13:12:25.777772 27122 heartbeater.cc:503] Master 127.8.88.190:45293 was elected leader, sending a full tablet report...
I20260928 13:12:25.777788 27141 consensus_queue.cc:260] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cc39c92baa460588c4a853779da34b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40181 } } peers { permanent_uuid: "3c6479b7b3cc4293b640e8be81143361" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37847 } } peers { permanent_uuid: "0c18584d04d148a6896f90826acdcfbb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39403 } }
I20260928 13:12:25.777932 27140 ts_tablet_manager.cc:1428] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:12:25.778141 26988 heartbeater.cc:503] Master 127.8.88.190:45293 was elected leader, sending a full tablet report...
I20260928 13:12:25.778187 27141 ts_tablet_manager.cc:1428] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 13:12:25.778263 26854 heartbeater.cc:503] Master 127.8.88.190:45293 was elected leader, sending a full tablet report...
W20260928 13:12:25.799798 26989 tablet.cc:2369] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:12:25.881095 26855 tablet.cc:2369] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:12:25.916363 27146 raft_consensus.cc:482] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:12:25.916491 27146 raft_consensus.cc:504] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cc39c92baa460588c4a853779da34b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40181 } } peers { permanent_uuid: "3c6479b7b3cc4293b640e8be81143361" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37847 } } peers { permanent_uuid: "0c18584d04d148a6896f90826acdcfbb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39403 } }
I20260928 13:12:25.916841 27146 leader_election.cc:288] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1cc39c92baa460588c4a853779da34b (127.8.88.131:40181), 0c18584d04d148a6896f90826acdcfbb (127.8.88.129:39403)
I20260928 13:12:25.920212 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.917262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:52719 (local address 127.8.88.129:39403)
0928 13:12:25.917322 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:25.917324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:25.917335 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:12:25.917373 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:25.917374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:25.917385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:25.917477 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:25.917689 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.918463 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.919452 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.919831 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.919926 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.919936 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:25.920094 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 13:12:25.920137 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 13:12:25.920238 26990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.917133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:39403 (local address 127.8.88.130:52719)
0928 13:12:25.917236 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:25.917248 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:25.917297 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:25.917516 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:25.917517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:25.917519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:25.917603 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.917605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.918559 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.918560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.919344 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.919346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.919925 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.919926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.920008 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.920011 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:12:25.920075 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 13:12:25.920170 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 13:12:25.920302 27135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.917261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:57949 (local address 127.8.88.131:40181)
0928 13:12:25.917314 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:25.917316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:25.917494 (+ 178us) server_negotiation.cc:407] Connection header received
0928 13:12:25.917526 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:25.917527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:25.917536 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:25.917601 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:25.917817 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.918540 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.919516 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.919904 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.920004 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.920013 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:25.920195 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 13:12:25.920233 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:12:25.920526 27148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.917133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:40181 (local address 127.8.88.130:57949)
0928 13:12:25.917403 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:25.917414 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:25.917477 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:25.917623 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:25.917623 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:25.917625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:25.917723 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.917725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.918609 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.918610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.919415 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:25.919417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.920008 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.920008 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:25.920091 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.920095 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:12:25.920182 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 13:12:25.920224 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":55,"thread_start_us":120,"threads_started":1}
I20260928 13:12:25.920660 26808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2eb1b5b4d0481aa7be54bfebb83371" candidate_uuid: "3c6479b7b3cc4293b640e8be81143361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c18584d04d148a6896f90826acdcfbb" is_pre_election: true
I20260928 13:12:25.920872 26808 raft_consensus.cc:2454] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6479b7b3cc4293b640e8be81143361 in term 0.
I20260928 13:12:25.920998 27076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2eb1b5b4d0481aa7be54bfebb83371" candidate_uuid: "3c6479b7b3cc4293b640e8be81143361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1cc39c92baa460588c4a853779da34b" is_pre_election: true
I20260928 13:12:25.921154 27076 raft_consensus.cc:2454] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c6479b7b3cc4293b640e8be81143361 in term 0.
I20260928 13:12:25.921183 26878 leader_election.cc:302] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c18584d04d148a6896f90826acdcfbb, 3c6479b7b3cc4293b640e8be81143361; no voters:
I20260928 13:12:25.921389 27146 raft_consensus.cc:2781] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:12:25.921468 27146 raft_consensus.cc:482] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:12:25.921486 27146 raft_consensus.cc:3037] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:25.922183 27146 raft_consensus.cc:504] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cc39c92baa460588c4a853779da34b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40181 } } peers { permanent_uuid: "3c6479b7b3cc4293b640e8be81143361" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37847 } } peers { permanent_uuid: "0c18584d04d148a6896f90826acdcfbb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39403 } }
I20260928 13:12:25.922334 27146 leader_election.cc:288] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [CANDIDATE]: Term 1 election: Requested vote from peers e1cc39c92baa460588c4a853779da34b (127.8.88.131:40181), 0c18584d04d148a6896f90826acdcfbb (127.8.88.129:39403)
I20260928 13:12:25.922488 27076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2eb1b5b4d0481aa7be54bfebb83371" candidate_uuid: "3c6479b7b3cc4293b640e8be81143361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1cc39c92baa460588c4a853779da34b"
I20260928 13:12:25.922544 26808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d2eb1b5b4d0481aa7be54bfebb83371" candidate_uuid: "3c6479b7b3cc4293b640e8be81143361" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c18584d04d148a6896f90826acdcfbb"
I20260928 13:12:25.922565 27076 raft_consensus.cc:3037] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:25.922632 26808 raft_consensus.cc:3037] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:25.923331 27076 raft_consensus.cc:2454] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6479b7b3cc4293b640e8be81143361 in term 1.
I20260928 13:12:25.923472 26808 raft_consensus.cc:2454] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c6479b7b3cc4293b640e8be81143361 in term 1.
I20260928 13:12:25.923516 26875 leader_election.cc:302] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c6479b7b3cc4293b640e8be81143361, e1cc39c92baa460588c4a853779da34b; no voters:
I20260928 13:12:25.923592 27146 raft_consensus.cc:2781] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:25.923756 27146 raft_consensus.cc:686] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [term 1 LEADER]: Becoming Leader. State: Replica: 3c6479b7b3cc4293b640e8be81143361, State: Running, Role: LEADER
I20260928 13:12:25.923866 27146 consensus_queue.cc:237] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cc39c92baa460588c4a853779da34b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40181 } } peers { permanent_uuid: "3c6479b7b3cc4293b640e8be81143361" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37847 } } peers { permanent_uuid: "0c18584d04d148a6896f90826acdcfbb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39403 } }
I20260928 13:12:25.924825 26662 catalog_manager.cc:5495] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c6479b7b3cc4293b640e8be81143361 (127.8.88.130). New cstate: current_term: 1 leader_uuid: "3c6479b7b3cc4293b640e8be81143361" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1cc39c92baa460588c4a853779da34b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c6479b7b3cc4293b640e8be81143361" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 37847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c18584d04d148a6896f90826acdcfbb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260928 13:12:25.952625 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.950077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.8.88.130:37847)
0928 13:12:25.950144 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:25.950146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:25.950156 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:12:25.950247 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:25.950249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:25.950276 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:25.950361 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:25.950498 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.950994 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.951811 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:25.952009 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:25.952111 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:25.952256 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:25.952362 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:25.952471 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:25.952502 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 13:12:25.952514 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 13:12:25.954702 26808 raft_consensus.cc:1261] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 0c18584d04d148a6896f90826acdcfbb [term 1 FOLLOWER]: Refusing update from remote peer 3c6479b7b3cc4293b640e8be81143361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:25.954746 27076 raft_consensus.cc:1261] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b [term 1 FOLLOWER]: Refusing update from remote peer 3c6479b7b3cc4293b640e8be81143361: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:25.955106 27149 consensus_queue.cc:1035] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c18584d04d148a6896f90826acdcfbb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:25.955263 27146 consensus_queue.cc:1035] T 7d2eb1b5b4d0481aa7be54bfebb83371 P 3c6479b7b3cc4293b640e8be81143361 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1cc39c92baa460588c4a853779da34b" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 40181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:25.957827 27154 mvcc.cc:205] Tried to move back new op lower bound from 7334302293827301376 to 7334302293704769536. Current Snapshot: MvccSnapshot[applied={T|T < 7334302293827301376}]
I20260928 13:12:25.958333 27153 mvcc.cc:205] Tried to move back new op lower bound from 7334302293827301376 to 7334302293704769536. Current Snapshot: MvccSnapshot[applied={T|T < 7334302293827301376}]
I20260928 13:12:25.960305 26662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:12:25.960388 26662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:12:25.961167 26662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f3790b9d2534cb2863d0cce247eea78: Sending DeleteTablet for 3 replicas of tablet 7d2eb1b5b4d0481aa7be54bfebb83371
I20260928 13:12:25.961490 27056 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2eb1b5b4d0481aa7be54bfebb83371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:25 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:35084
I20260928 13:12:25.961549 26922 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2eb1b5b4d0481aa7be54bfebb83371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:25 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:48860
I20260928 13:12:25.961760 27162 tablet_replica.cc:330] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b: stopping tablet replica
I20260928 13:12:25.961751 26788 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d2eb1b5b4d0481aa7be54bfebb83371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:25 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:56962
I20260928 13:12:25.961861 27162 raft_consensus.cc:2229] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:12:25.961954 27162 raft_consensus.cc:2258] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:12:25.962009 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 26725
I20260928 13:12:25.964066 27162 ts_tablet_manager.cc:1905] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:12:25.965987 27162 ts_tablet_manager.cc:1918] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:12:25.966066 27162 log.cc:1196] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/7d2eb1b5b4d0481aa7be54bfebb83371
I20260928 13:12:25.966326 27162 ts_tablet_manager.cc:1939] T 7d2eb1b5b4d0481aa7be54bfebb83371 P e1cc39c92baa460588c4a853779da34b: Deleting consensus metadata
I20260928 13:12:25.966925 26647 catalog_manager.cc:4841] TS e1cc39c92baa460588c4a853779da34b (127.8.88.131:40181): tablet 7d2eb1b5b4d0481aa7be54bfebb83371 (table test-table [id=62a4a309a1f340998e52d95c52def925]) successfully deleted
W20260928 13:12:25.967082 26650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:39403 (error 108)
W20260928 13:12:25.968093 27137 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:25.967476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:39403 (local address 127.0.0.1:56966)
0928 13:12:25.967542 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:25.967553 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:25.967594 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:25.968049 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:39403: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 13:12:25.968256 26650 catalog_manager.cc:4584] TS 0c18584d04d148a6896f90826acdcfbb (127.8.88.129:39403): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d2eb1b5b4d0481aa7be54bfebb83371: Network error: Client connection negotiation failed: client connection to 127.8.88.129:39403: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 13:12:25.968339 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 26859
I20260928 13:12:25.974448 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 26993
I20260928 13:12:25.974632 27137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:25.974448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:37847 (local address 127.0.0.1:48882)
0928 13:12:25.974514 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:25.974590 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.130:37847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 13:12:25.979065 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 26631
2026-09-28T13:12:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2142 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T13:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:26Z Disabled control of system clock
I20260928 13:12:26.009300 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46843
--webserver_interface=127.8.88.190
--webserver_port=0
--builtin_ntp_servers=127.8.88.148:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:12:26.093779 27169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:26.094013 27169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:26.094043 27169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:26.095408 27169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:26.095476 27169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:26.095491 27169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:26.095508 27169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:26.095520 27169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:26.096995 27169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:26.097218 27169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:26.097469 27169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:26.102526 27175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:26.102535 27174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:26.102887 27177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:26.102846 27169 server_base.cc:1034] running on GCE node
I20260928 13:12:26.103159 27169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:26.103470 27169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:26.104631 27169 hybrid_clock.cc:648] HybridClock initialized: now 1790601146104619 us; error 33 us; skew 500 ppm
I20260928 13:12:26.106006 27169 webserver.cc:460] Webserver started at http://127.8.88.190:44729/ using document root <none> and password file <none>
I20260928 13:12:26.106287 27169 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:26.106362 27169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:26.106514 27169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:26.107633 27169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "37d9dca6c5814cadaba2d7af645aa3a1"
format_stamp: "Formatted at 2026-09-28 13:12:26 on dist-test-slave-p1lc"
server_key: "a99b31309af6b194c90cf4d745c3363e"
server_key_iv: "1c4c6b509128a393928309855224749d"
server_key_version: "encryptionkey@0"
I20260928 13:12:26.108014 27169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "37d9dca6c5814cadaba2d7af645aa3a1"
format_stamp: "Formatted at 2026-09-28 13:12:26 on dist-test-slave-p1lc"
server_key: "a99b31309af6b194c90cf4d745c3363e"
server_key_iv: "1c4c6b509128a393928309855224749d"
server_key_version: "encryptionkey@0"
I20260928 13:12:26.109483 27169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:26.110325 27183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:26.110526 27169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:26.110631 27169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "37d9dca6c5814cadaba2d7af645aa3a1"
format_stamp: "Formatted at 2026-09-28 13:12:26 on dist-test-slave-p1lc"
server_key: "a99b31309af6b194c90cf4d745c3363e"
server_key_iv: "1c4c6b509128a393928309855224749d"
server_key_version: "encryptionkey@0"
I20260928 13:12:26.110735 27169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:26.139436 27169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:26.139833 27169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:26.140007 27169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:26.144243 27169 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:46843
I20260928 13:12:26.144284 27235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:46843 every 8 connection(s)
I20260928 13:12:26.144622 27169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:26.145603 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:26.147933 27236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1: Bootstrap starting.
I20260928 13:12:26.148479 27236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:26.148767 27236 log.cc:824] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:26.149693 27236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1: No bootstrap required, opened a new log
I20260928 13:12:26.152268 27236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d9dca6c5814cadaba2d7af645aa3a1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46843 } }
I20260928 13:12:26.152474 27236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:26.152503 27236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d9dca6c5814cadaba2d7af645aa3a1, State: Initialized, Role: FOLLOWER
I20260928 13:12:26.152649 27236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d9dca6c5814cadaba2d7af645aa3a1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46843 } }
I20260928 13:12:26.152733 27236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:26.152756 27236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:26.152818 27236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:26.153417 27236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d9dca6c5814cadaba2d7af645aa3a1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46843 } }
I20260928 13:12:26.153506 27236 leader_election.cc:302] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37d9dca6c5814cadaba2d7af645aa3a1; no voters:
I20260928 13:12:26.153661 27236 leader_election.cc:288] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:26.153786 27239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:26.153910 27236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:26.153978 27239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 37d9dca6c5814cadaba2d7af645aa3a1, State: Running, Role: LEADER
I20260928 13:12:26.154124 27239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d9dca6c5814cadaba2d7af645aa3a1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46843 } }
I20260928 13:12:26.154302 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 27169
I20260928 13:12:26.154402 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:26.154628 8546 external_mini_cluster.cc:1428] Setting key 83b11b1ab0dc9bbee326defd6fe91c14
I20260928 13:12:26.154960 27241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d9dca6c5814cadaba2d7af645aa3a1. Latest consensus state: current_term: 1 leader_uuid: "37d9dca6c5814cadaba2d7af645aa3a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d9dca6c5814cadaba2d7af645aa3a1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46843 } } }
I20260928 13:12:26.155050 27241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:26.155282 27249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:26.155673 27240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37d9dca6c5814cadaba2d7af645aa3a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d9dca6c5814cadaba2d7af645aa3a1" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46843 } } }
I20260928 13:12:26.155767 27240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:26.155946 27249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:26.158026 27249 catalog_manager.cc:1347] Generated new cluster ID: d0cd279178564b96ac1e88cfb21de0d0
I20260928 13:12:26.158083 27249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:26.161769 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.155324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42492 (local address 127.8.88.190:46843)
0928 13:12:26.157023 (+ 1699us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:26.157033 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:26.157064 (+ 31us) server_negotiation.cc:407] Connection header received
0928 13:12:26.157176 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:26.157183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:26.157366 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:26.157518 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:26.157744 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.158880 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.159692 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.160123 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.160258 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.161008 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:26.161012 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:26.161012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:26.161203 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:26.161236 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:26.161247 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:26.161321 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:26.161403 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 13:12:26.161586 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1622,"thread_start_us":83,"threads_started":1}
I20260928 13:12:26.170691 27249 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:26.171223 27249 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:26.181973 27249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1: Generated new TSK 0
I20260928 13:12:26.182228 27249 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:26.185009 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46843
--builtin_ntp_servers=127.8.88.148:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:12:26.269778 27260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:26.269989 27260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:26.270012 27260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:26.271265 27260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:26.271313 27260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:26.271341 27260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:12:26.272755 27260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:26.273110 27260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:26.273452 27260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:26.280119 27266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:26.280244 27268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:26.280088 27265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:26.280141 27260 server_base.cc:1034] running on GCE node
I20260928 13:12:26.280529 27260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:26.280848 27260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:26.282063 27260 hybrid_clock.cc:648] HybridClock initialized: now 1790601146282049 us; error 39 us; skew 500 ppm
I20260928 13:12:26.283557 27260 webserver.cc:460] Webserver started at http://127.8.88.129:40981/ using document root <none> and password file <none>
I20260928 13:12:26.283845 27260 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:26.283897 27260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:26.284031 27260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:26.285213 27260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "15d94ec465ae469b97141e3ee8c45814"
format_stamp: "Formatted at 2026-09-28 13:12:26 on dist-test-slave-p1lc"
server_key: "a0d59876b26906ea90d0acc7c1ffa995"
server_key_iv: "dce22b1378a4ff62b928c4370ed4a2b7"
server_key_version: "encryptionkey@0"
I20260928 13:12:26.285588 27260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "15d94ec465ae469b97141e3ee8c45814"
format_stamp: "Formatted at 2026-09-28 13:12:26 on dist-test-slave-p1lc"
server_key: "a0d59876b26906ea90d0acc7c1ffa995"
server_key_iv: "dce22b1378a4ff62b928c4370ed4a2b7"
server_key_version: "encryptionkey@0"
I20260928 13:12:26.287060 27260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:12:26.287905 27274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:26.288106 27260 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:26.288218 27260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "15d94ec465ae469b97141e3ee8c45814"
format_stamp: "Formatted at 2026-09-28 13:12:26 on dist-test-slave-p1lc"
server_key: "a0d59876b26906ea90d0acc7c1ffa995"
server_key_iv: "dce22b1378a4ff62b928c4370ed4a2b7"
server_key_version: "encryptionkey@0"
I20260928 13:12:26.288323 27260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:26.298033 27260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:26.298400 27260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:26.298571 27260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:26.298822 27260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:26.299160 27260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:26.299216 27260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:26.299291 27260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:26.299322 27260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:26.305636 27260 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:39539
I20260928 13:12:26.305680 27387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:39539 every 8 connection(s)
I20260928 13:12:26.306046 27260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:12:26.310328 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 27260
I20260928 13:12:26.310429 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:12:26.310626 8546 external_mini_cluster.cc:1428] Setting key 8affb25c98432cc0bafa86edebd583bf
I20260928 13:12:26.311956 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46843
--builtin_ntp_servers=127.8.88.148:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 13:12:26.312943 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.306867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:55841 (local address 127.8.88.190:46843)
0928 13:12:26.306939 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:26.306942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:26.307222 (+ 280us) server_negotiation.cc:407] Connection header received
0928 13:12:26.307727 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:26.307728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:26.307755 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:26.307815 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:26.308522 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.309146 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.310808 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.311155 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.311236 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.312517 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:26.312519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:26.312520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:26.312525 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:26.312540 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:26.312545 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:26.312629 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:26.312759 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 13:12:26.312818 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 13:12:26.313153 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.306858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46843 (local address 127.8.88.129:55841)
0928 13:12:26.307086 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:26.307123 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:26.307639 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:26.307900 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:26.307904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:26.307913 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:26.308448 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.308452 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.309249 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.309252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.310607 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.310610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.311240 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.311242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.311489 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.312321 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:12:26.312322 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:26.312380 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:12:26.312415 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:26.312653 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:26.312659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:26.312663 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:12:26.312751 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 13:12:26.312901 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":119,"threads_started":1}
I20260928 13:12:26.314004 27388 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46843
I20260928 13:12:26.314114 27388 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:26.314348 27388 heartbeater.cc:511] Master 127.8.88.190:46843 requested a full tablet report, sending...
I20260928 13:12:26.314939 27200 ts_manager.cc:194] Registered new tserver with Master: 15d94ec465ae469b97141e3ee8c45814 (127.8.88.129:39539)
I20260928 13:12:26.315907 27200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.88.129:55841
W20260928 13:12:26.398716 27391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:26.398967 27391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:26.399003 27391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:26.400213 27391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:26.400282 27391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:26.400322 27391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:12:26.401752 27391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:26.402029 27391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:26.402344 27391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:26.407847 27397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:26.407847 27399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:26.407867 27396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:26.408335 27391 server_base.cc:1034] running on GCE node
I20260928 13:12:26.408576 27391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:26.408901 27391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:26.410075 27391 hybrid_clock.cc:648] HybridClock initialized: now 1790601146410061 us; error 29 us; skew 500 ppm
I20260928 13:12:26.411437 27391 webserver.cc:460] Webserver started at http://127.8.88.130:46309/ using document root <none> and password file <none>
I20260928 13:12:26.411691 27391 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:26.411813 27391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:26.411960 27391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:26.413465 27391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "7306d6f005e348faa1bbdbae78cbe0e0"
format_stamp: "Formatted at 2026-09-28 13:12:26 on dist-test-slave-p1lc"
server_key: "9d3d8b2128fada4bbb171e5e1fee6dc3"
server_key_iv: "6838407b338e19066b805f654f296e2e"
server_key_version: "encryptionkey@0"
I20260928 13:12:26.413839 27391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "7306d6f005e348faa1bbdbae78cbe0e0"
format_stamp: "Formatted at 2026-09-28 13:12:26 on dist-test-slave-p1lc"
server_key: "9d3d8b2128fada4bbb171e5e1fee6dc3"
server_key_iv: "6838407b338e19066b805f654f296e2e"
server_key_version: "encryptionkey@0"
I20260928 13:12:26.415275 27391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:12:26.416049 27405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:26.416246 27391 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:12:26.416347 27391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "7306d6f005e348faa1bbdbae78cbe0e0"
format_stamp: "Formatted at 2026-09-28 13:12:26 on dist-test-slave-p1lc"
server_key: "9d3d8b2128fada4bbb171e5e1fee6dc3"
server_key_iv: "6838407b338e19066b805f654f296e2e"
server_key_version: "encryptionkey@0"
I20260928 13:12:26.416436 27391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:26.430514 27391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:26.430852 27391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:26.431011 27391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:26.431252 27391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:26.431613 27391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:26.431667 27391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:26.431702 27391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:26.431730 27391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:26.437886 27391 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:35819
I20260928 13:12:26.437990 27518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:35819 every 8 connection(s)
I20260928 13:12:26.438339 27391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:12:26.438581 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 27391
I20260928 13:12:26.438681 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:12:26.438903 8546 external_mini_cluster.cc:1428] Setting key b717a10b02d0f061913d347435c447e9
I20260928 13:12:26.440090 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46843
--builtin_ntp_servers=127.8.88.148:41035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 13:12:26.445221 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.439392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:49959 (local address 127.8.88.190:46843)
0928 13:12:26.439452 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:26.439454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:26.439682 (+ 228us) server_negotiation.cc:407] Connection header received
0928 13:12:26.440497 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:26.440498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:26.440510 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:26.440565 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:26.441552 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.442403 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.443432 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.443771 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.443852 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.444837 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:26.444838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:26.444839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:26.444845 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:26.444863 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:26.444889 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:26.444963 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:26.445043 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 13:12:26.445098 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:12:26.445317 27521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.439285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46843 (local address 127.8.88.130:49959)
0928 13:12:26.439544 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:26.439594 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:26.440386 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:26.440641 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:26.440646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:26.440657 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:26.441403 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.441411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.442503 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.442504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.443344 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.443347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.443886 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.443887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.444070 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.444687 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:12:26.444688 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:26.444718 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:12:26.444743 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:26.444964 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:26.444967 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:26.444970 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:12:26.445023 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 13:12:26.445119 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":139,"threads_started":1}
I20260928 13:12:26.445858 27519 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46843
I20260928 13:12:26.445976 27519 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:26.446168 27519 heartbeater.cc:511] Master 127.8.88.190:46843 requested a full tablet report, sending...
I20260928 13:12:26.446599 27200 ts_manager.cc:194] Registered new tserver with Master: 7306d6f005e348faa1bbdbae78cbe0e0 (127.8.88.130:35819)
I20260928 13:12:26.447158 27200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.88.130:49959
W20260928 13:12:26.524925 27522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:26.525211 27522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:26.525238 27522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:26.526479 27522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:26.526521 27522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:26.526546 27522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:12:26.527912 27522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:41035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:46843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:26.528190 27522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:26.528509 27522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:26.533778 27527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:26.533771 27530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:26.533771 27528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:26.533790 27522 server_base.cc:1034] running on GCE node
I20260928 13:12:26.534229 27522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:26.534442 27522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:26.535688 27522 hybrid_clock.cc:648] HybridClock initialized: now 1790601146535671 us; error 30 us; skew 500 ppm
I20260928 13:12:26.537096 27522 webserver.cc:460] Webserver started at http://127.8.88.131:42011/ using document root <none> and password file <none>
I20260928 13:12:26.537350 27522 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:26.537423 27522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:26.537580 27522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:26.538702 27522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "7b7327667b5a49b69a3af67c0564051e"
format_stamp: "Formatted at 2026-09-28 13:12:26 on dist-test-slave-p1lc"
server_key: "164a8233ae504c10199a1b9ec6e2c546"
server_key_iv: "d89569ccac0dbb86eda5f749b8cfe515"
server_key_version: "encryptionkey@0"
I20260928 13:12:26.539069 27522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "7b7327667b5a49b69a3af67c0564051e"
format_stamp: "Formatted at 2026-09-28 13:12:26 on dist-test-slave-p1lc"
server_key: "164a8233ae504c10199a1b9ec6e2c546"
server_key_iv: "d89569ccac0dbb86eda5f749b8cfe515"
server_key_version: "encryptionkey@0"
I20260928 13:12:26.540429 27522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:26.541308 27536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:26.541481 27522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:26.541601 27522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "7b7327667b5a49b69a3af67c0564051e"
format_stamp: "Formatted at 2026-09-28 13:12:26 on dist-test-slave-p1lc"
server_key: "164a8233ae504c10199a1b9ec6e2c546"
server_key_iv: "d89569ccac0dbb86eda5f749b8cfe515"
server_key_version: "encryptionkey@0"
I20260928 13:12:26.541685 27522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:26.552172 27522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:26.552523 27522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:26.552690 27522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:26.553030 27522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:26.553416 27522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:26.553462 27522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:26.553499 27522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:26.553529 27522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:26.560089 27522 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:46145
I20260928 13:12:26.560139 27649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:46145 every 8 connection(s)
I20260928 13:12:26.560487 27522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:12:26.566249 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 27522
I20260928 13:12:26.566381 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:12:26.566566 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.561309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:37169 (local address 127.8.88.190:46843)
0928 13:12:26.561366 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:26.561369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:26.561674 (+ 305us) server_negotiation.cc:407] Connection header received
0928 13:12:26.562417 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:26.562418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:26.562428 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:26.562482 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:26.563162 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.563675 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.564951 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.565173 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.565238 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.566208 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:26.566210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:26.566210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:26.566215 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:26.566230 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:26.566235 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:26.566290 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:26.566438 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 13:12:26.566474 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 13:12:26.566601 8546 external_mini_cluster.cc:1428] Setting key 3c60a819847a663a33b031b4ecc8ef6c
I20260928 13:12:26.566843 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.561294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46843 (local address 127.8.88.131:37169)
0928 13:12:26.561495 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:26.561561 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:26.562321 (+ 760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:26.562538 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:26.562541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:26.562549 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:26.563065 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.563069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.563919 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.563920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.564802 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.564804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.565256 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.565257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.565374 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.566014 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:12:26.566015 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:26.566059 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:12:26.566089 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:26.566311 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:26.566318 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:26.566322 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:12:26.566406 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 13:12:26.566525 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260928 13:12:26.567276 27650 heartbeater.cc:347] Connected to a master server at 127.8.88.190:46843
I20260928 13:12:26.567395 27650 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:26.567591 27650 heartbeater.cc:511] Master 127.8.88.190:46843 requested a full tablet report, sending...
I20260928 13:12:26.568042 27200 ts_manager.cc:194] Registered new tserver with Master: 7b7327667b5a49b69a3af67c0564051e (127.8.88.131:46145)
I20260928 13:12:26.568470 27200 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.88.131:37169
I20260928 13:12:26.579149 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:12:26.583789 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.580638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.8.88.190:46843)
0928 13:12:26.580728 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:26.580730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:26.580811 (+ 81us) server_negotiation.cc:407] Connection header received
0928 13:12:26.580851 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:26.580851 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:26.580877 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:26.580926 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:26.581084 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.581579 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.582372 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.582624 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.582689 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.583415 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:26.583416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:26.583416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:26.583421 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:26.583436 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:26.583440 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:26.583521 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:26.583681 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 13:12:26.583696 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 13:12:26.585758 27200 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:12:26.587347 27200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:12:26.594252 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.590070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:46145 (local address 127.0.0.1:41286)
0928 13:12:26.590378 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:26.590389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:26.590431 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:26.590624 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:26.590627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:26.590629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:26.590816 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.590820 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.591969 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.591970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.593163 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.593166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.593945 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.593946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.594023 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.594036 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 13:12:26.594112 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 13:12:26.594161 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":51,"thread_start_us":68,"threads_started":1}
I20260928 13:12:26.594264 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.590195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.8.88.131:46145)
0928 13:12:26.590409 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:26.590413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:26.590424 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:12:26.590497 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:26.590499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:26.590515 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:26.590597 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:26.590966 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.591806 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.593252 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.593849 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.593927 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.594019 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:26.594140 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 13:12:26.594180 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
I20260928 13:12:26.594771 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.589867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:35819 (local address 127.0.0.1:39632)
0928 13:12:26.590211 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:26.590278 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:26.590364 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:26.590732 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:26.590733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:26.590734 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:26.590764 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.590765 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.592343 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.592344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.593599 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.593603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.594496 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.594497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.594575 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.594579 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:12:26.594639 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 13:12:26.594689 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":35,"thread_start_us":110,"threads_started":1}
I20260928 13:12:26.594810 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.589957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39632 (local address 127.8.88.130:35819)
0928 13:12:26.590109 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:26.590115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:26.590449 (+ 334us) server_negotiation.cc:407] Connection header received
0928 13:12:26.590497 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:26.590499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:26.590515 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:26.590597 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:26.590850 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.592082 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.593704 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.594397 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.594476 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.594563 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:26.594668 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 13:12:26.594713 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260928 13:12:26.594975 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.589942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.8.88.129:39539)
0928 13:12:26.590205 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:26.590208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:26.590322 (+ 114us) server_negotiation.cc:407] Connection header received
0928 13:12:26.590464 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:26.590467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:26.590485 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:26.590535 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:26.590859 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.592242 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.593701 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.594483 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.594567 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.594657 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:26.594841 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 13:12:26.594890 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":85,"threads_started":1}
I20260928 13:12:26.595218 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.589845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:39539 (local address 127.0.0.1:34474)
0928 13:12:26.590199 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:26.590243 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:26.590334 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:26.590662 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:26.590663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:26.590665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:26.590774 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.590776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.592343 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.592344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.593603 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.593604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.594691 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.594692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.594748 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.594750 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 13:12:26.595095 (+ 345us) client_negotiation.cc:240] Negotiation successful
0928 13:12:26.595140 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":63,"thread_start_us":79,"threads_started":1}
I20260928 13:12:26.595484 27453 tablet_service.cc:1467] Processing CreateTablet for tablet c7e6544e939844d68b061624bed7130d (DEFAULT_TABLE table=test-table [id=ff76150053ee4fdab329c9d15b7e7d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:26.595717 27584 tablet_service.cc:1467] Processing CreateTablet for tablet c7e6544e939844d68b061624bed7130d (DEFAULT_TABLE table=test-table [id=ff76150053ee4fdab329c9d15b7e7d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:26.595918 27453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e6544e939844d68b061624bed7130d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:26.596017 27584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e6544e939844d68b061624bed7130d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:26.595974 27322 tablet_service.cc:1467] Processing CreateTablet for tablet c7e6544e939844d68b061624bed7130d (DEFAULT_TABLE table=test-table [id=ff76150053ee4fdab329c9d15b7e7d85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:26.596311 27322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e6544e939844d68b061624bed7130d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:26.598763 27664 tablet_bootstrap.cc:492] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e: Bootstrap starting.
I20260928 13:12:26.599370 27665 tablet_bootstrap.cc:492] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0: Bootstrap starting.
I20260928 13:12:26.599421 27664 tablet_bootstrap.cc:654] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:26.599681 27666 tablet_bootstrap.cc:492] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814: Bootstrap starting.
I20260928 13:12:26.599728 27664 log.cc:824] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:26.600268 27665 tablet_bootstrap.cc:654] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:26.600405 27666 tablet_bootstrap.cc:654] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:26.600574 27664 tablet_bootstrap.cc:492] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e: No bootstrap required, opened a new log
I20260928 13:12:26.600608 27665 log.cc:824] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:26.600646 27664 ts_tablet_manager.cc:1397] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:12:26.600728 27666 log.cc:824] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:26.601418 27665 tablet_bootstrap.cc:492] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0: No bootstrap required, opened a new log
I20260928 13:12:26.601500 27665 ts_tablet_manager.cc:1397] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:12:26.602738 27664 raft_consensus.cc:348] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d94ec465ae469b97141e3ee8c45814" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39539 } } peers { permanent_uuid: "7b7327667b5a49b69a3af67c0564051e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46145 } } peers { permanent_uuid: "7306d6f005e348faa1bbdbae78cbe0e0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 35819 } }
I20260928 13:12:26.602907 27664 raft_consensus.cc:374] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:26.602953 27664 raft_consensus.cc:729] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7327667b5a49b69a3af67c0564051e, State: Initialized, Role: FOLLOWER
I20260928 13:12:26.603094 27664 consensus_queue.cc:260] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d94ec465ae469b97141e3ee8c45814" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39539 } } peers { permanent_uuid: "7b7327667b5a49b69a3af67c0564051e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46145 } } peers { permanent_uuid: "7306d6f005e348faa1bbdbae78cbe0e0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 35819 } }
I20260928 13:12:26.603394 27664 ts_tablet_manager.cc:1428] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:12:26.603448 27650 heartbeater.cc:503] Master 127.8.88.190:46843 was elected leader, sending a full tablet report...
I20260928 13:12:26.603633 27665 raft_consensus.cc:348] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d94ec465ae469b97141e3ee8c45814" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39539 } } peers { permanent_uuid: "7b7327667b5a49b69a3af67c0564051e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46145 } } peers { permanent_uuid: "7306d6f005e348faa1bbdbae78cbe0e0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 35819 } }
I20260928 13:12:26.603819 27665 raft_consensus.cc:374] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:26.603860 27665 raft_consensus.cc:729] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7306d6f005e348faa1bbdbae78cbe0e0, State: Initialized, Role: FOLLOWER
I20260928 13:12:26.603969 27665 consensus_queue.cc:260] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d94ec465ae469b97141e3ee8c45814" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39539 } } peers { permanent_uuid: "7b7327667b5a49b69a3af67c0564051e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46145 } } peers { permanent_uuid: "7306d6f005e348faa1bbdbae78cbe0e0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 35819 } }
I20260928 13:12:26.604254 27665 ts_tablet_manager.cc:1428] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 13:12:26.604282 27519 heartbeater.cc:503] Master 127.8.88.190:46843 was elected leader, sending a full tablet report...
I20260928 13:12:26.604693 27666 tablet_bootstrap.cc:492] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814: No bootstrap required, opened a new log
I20260928 13:12:26.604769 27666 ts_tablet_manager.cc:1397] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 13:12:26.606753 27666 raft_consensus.cc:348] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d94ec465ae469b97141e3ee8c45814" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39539 } } peers { permanent_uuid: "7b7327667b5a49b69a3af67c0564051e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46145 } } peers { permanent_uuid: "7306d6f005e348faa1bbdbae78cbe0e0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 35819 } }
I20260928 13:12:26.606886 27666 raft_consensus.cc:374] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:26.606907 27666 raft_consensus.cc:729] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d94ec465ae469b97141e3ee8c45814, State: Initialized, Role: FOLLOWER
I20260928 13:12:26.607020 27666 consensus_queue.cc:260] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d94ec465ae469b97141e3ee8c45814" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39539 } } peers { permanent_uuid: "7b7327667b5a49b69a3af67c0564051e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46145 } } peers { permanent_uuid: "7306d6f005e348faa1bbdbae78cbe0e0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 35819 } }
I20260928 13:12:26.607308 27666 ts_tablet_manager.cc:1428] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 13:12:26.607431 27388 heartbeater.cc:503] Master 127.8.88.190:46843 was elected leader, sending a full tablet report...
I20260928 13:12:26.657697 27672 raft_consensus.cc:482] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:12:26.657835 27672 raft_consensus.cc:504] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d94ec465ae469b97141e3ee8c45814" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39539 } } peers { permanent_uuid: "7b7327667b5a49b69a3af67c0564051e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46145 } } peers { permanent_uuid: "7306d6f005e348faa1bbdbae78cbe0e0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 35819 } }
I20260928 13:12:26.658313 27672 leader_election.cc:288] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b7327667b5a49b69a3af67c0564051e (127.8.88.131:46145), 7306d6f005e348faa1bbdbae78cbe0e0 (127.8.88.130:35819)
I20260928 13:12:26.662287 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.658548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:35841 (local address 127.8.88.131:46145)
0928 13:12:26.658603 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:26.658605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:26.658615 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:12:26.658719 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:26.658720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:26.658778 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:26.658847 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:26.659058 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.659827 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.661336 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.661910 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.661986 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.661995 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:26.662179 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 13:12:26.662219 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 13:12:26.662314 27661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.658587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:59679 (local address 127.8.88.130:35819)
0928 13:12:26.658635 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:26.658638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:26.658766 (+ 128us) server_negotiation.cc:407] Connection header received
0928 13:12:26.658826 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:26.658828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:26.658837 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:26.658898 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:26.659057 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.659824 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.661336 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.661905 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.661986 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.661995 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:26.662179 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 13:12:26.662219 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:12:26.662323 27673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.658506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:35819 (local address 127.8.88.129:59679)
0928 13:12:26.658687 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:26.658696 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:26.658777 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:26.658911 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:26.658911 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:26.658912 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:26.658971 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.658974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.659922 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.659923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.661243 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.661246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.662001 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.662002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.662089 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.662091 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 13:12:26.662151 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 13:12:26.662202 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":101,"thread_start_us":92,"threads_started":1}
I20260928 13:12:26.662323 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.658483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:46145 (local address 127.8.88.129:35841)
0928 13:12:26.658568 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:26.658580 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:26.658629 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:26.658871 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:26.658889 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:26.658891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:26.658986 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.658988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.659922 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.659923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.661250 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:26.661251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.662001 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.662002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:26.662084 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.662089 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:12:26.662151 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 13:12:26.662202 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":29}
I20260928 13:12:26.662889 27473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e6544e939844d68b061624bed7130d" candidate_uuid: "15d94ec465ae469b97141e3ee8c45814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7306d6f005e348faa1bbdbae78cbe0e0" is_pre_election: true
I20260928 13:12:26.662976 27604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e6544e939844d68b061624bed7130d" candidate_uuid: "15d94ec465ae469b97141e3ee8c45814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7327667b5a49b69a3af67c0564051e" is_pre_election: true
I20260928 13:12:26.663069 27473 raft_consensus.cc:2454] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15d94ec465ae469b97141e3ee8c45814 in term 0.
I20260928 13:12:26.663142 27604 raft_consensus.cc:2454] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15d94ec465ae469b97141e3ee8c45814 in term 0.
I20260928 13:12:26.663342 27275 leader_election.cc:302] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d94ec465ae469b97141e3ee8c45814, 7306d6f005e348faa1bbdbae78cbe0e0; no voters:
I20260928 13:12:26.663497 27672 raft_consensus.cc:2781] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:12:26.663554 27672 raft_consensus.cc:482] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:12:26.663583 27672 raft_consensus.cc:3037] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:26.664255 27672 raft_consensus.cc:504] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d94ec465ae469b97141e3ee8c45814" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39539 } } peers { permanent_uuid: "7b7327667b5a49b69a3af67c0564051e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46145 } } peers { permanent_uuid: "7306d6f005e348faa1bbdbae78cbe0e0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 35819 } }
I20260928 13:12:26.664398 27672 leader_election.cc:288] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [CANDIDATE]: Term 1 election: Requested vote from peers 7b7327667b5a49b69a3af67c0564051e (127.8.88.131:46145), 7306d6f005e348faa1bbdbae78cbe0e0 (127.8.88.130:35819)
I20260928 13:12:26.664563 27604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e6544e939844d68b061624bed7130d" candidate_uuid: "15d94ec465ae469b97141e3ee8c45814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b7327667b5a49b69a3af67c0564051e"
I20260928 13:12:26.664626 27473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e6544e939844d68b061624bed7130d" candidate_uuid: "15d94ec465ae469b97141e3ee8c45814" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7306d6f005e348faa1bbdbae78cbe0e0"
I20260928 13:12:26.664672 27604 raft_consensus.cc:3037] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:26.664726 27473 raft_consensus.cc:3037] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:26.665475 27604 raft_consensus.cc:2454] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15d94ec465ae469b97141e3ee8c45814 in term 1.
I20260928 13:12:26.665510 27473 raft_consensus.cc:2454] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15d94ec465ae469b97141e3ee8c45814 in term 1.
I20260928 13:12:26.665688 27278 leader_election.cc:302] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d94ec465ae469b97141e3ee8c45814, 7b7327667b5a49b69a3af67c0564051e; no voters:
I20260928 13:12:26.665881 27672 raft_consensus.cc:2781] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:26.666069 27672 raft_consensus.cc:686] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [term 1 LEADER]: Becoming Leader. State: Replica: 15d94ec465ae469b97141e3ee8c45814, State: Running, Role: LEADER
I20260928 13:12:26.666179 27672 consensus_queue.cc:237] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d94ec465ae469b97141e3ee8c45814" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39539 } } peers { permanent_uuid: "7b7327667b5a49b69a3af67c0564051e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46145 } } peers { permanent_uuid: "7306d6f005e348faa1bbdbae78cbe0e0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 35819 } }
I20260928 13:12:26.667006 27199 catalog_manager.cc:5495] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15d94ec465ae469b97141e3ee8c45814 (127.8.88.129). New cstate: current_term: 1 leader_uuid: "15d94ec465ae469b97141e3ee8c45814" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d94ec465ae469b97141e3ee8c45814" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 39539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b7327667b5a49b69a3af67c0564051e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7306d6f005e348faa1bbdbae78cbe0e0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 35819 } health_report { overall_health: UNKNOWN } } }
W20260928 13:12:26.689023 27520 tablet.cc:2369] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:12:26.689348 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.686800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.8.88.129:39539)
0928 13:12:26.686871 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:26.686873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:26.686936 (+ 63us) server_negotiation.cc:407] Connection header received
0928 13:12:26.686977 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:26.686978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:26.686999 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:26.687054 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:26.687191 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.687666 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.688452 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:26.688641 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:26.688696 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:26.688847 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:26.689019 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:26.689101 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:26.689218 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 13:12:26.689230 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 13:12:26.691394 27473 raft_consensus.cc:1261] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0 [term 1 FOLLOWER]: Refusing update from remote peer 15d94ec465ae469b97141e3ee8c45814: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:26.691394 27604 raft_consensus.cc:1261] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e [term 1 FOLLOWER]: Refusing update from remote peer 15d94ec465ae469b97141e3ee8c45814: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:26.691834 27674 consensus_queue.cc:1035] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7306d6f005e348faa1bbdbae78cbe0e0" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 35819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:26.691983 27672 consensus_queue.cc:1035] T c7e6544e939844d68b061624bed7130d P 15d94ec465ae469b97141e3ee8c45814 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b7327667b5a49b69a3af67c0564051e" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 46145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:26.694536 27678 mvcc.cc:205] Tried to move back new op lower bound from 7334302296844578816 to 7334302296745385984. Current Snapshot: MvccSnapshot[applied={T|T < 7334302296844578816}]
I20260928 13:12:26.697714 27200 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:12:26.697837 27200 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:12:26.698634 27200 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37d9dca6c5814cadaba2d7af645aa3a1: Sending DeleteTablet for 3 replicas of tablet c7e6544e939844d68b061624bed7130d
I20260928 13:12:26.699015 27322 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e6544e939844d68b061624bed7130d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:26 UTC) from {username='slave'} at 127.0.0.1:34474
I20260928 13:12:26.699092 27453 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e6544e939844d68b061624bed7130d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:26 UTC) from {username='slave'} at 127.0.0.1:39632
I20260928 13:12:26.699288 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 27260
I20260928 13:12:26.699350 27584 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e6544e939844d68b061624bed7130d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:26 UTC) from {username='slave'} at 127.0.0.1:41286
I20260928 13:12:26.700695 27687 tablet_replica.cc:330] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0: stopping tablet replica
I20260928 13:12:26.700834 27687 raft_consensus.cc:2229] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:12:26.701015 27687 raft_consensus.cc:2258] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:12:26.701557 27687 ts_tablet_manager.cc:1905] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:12:26.703567 27687 ts_tablet_manager.cc:1918] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:12:26.703649 27687 log.cc:1196] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/c7e6544e939844d68b061624bed7130d
I20260928 13:12:26.703897 27687 ts_tablet_manager.cc:1939] T c7e6544e939844d68b061624bed7130d P 7306d6f005e348faa1bbdbae78cbe0e0: Deleting consensus metadata
I20260928 13:12:26.704434 27184 catalog_manager.cc:4841] TS 7306d6f005e348faa1bbdbae78cbe0e0 (127.8.88.130:35819): tablet c7e6544e939844d68b061624bed7130d (table test-table [id=ff76150053ee4fdab329c9d15b7e7d85]) successfully deleted
I20260928 13:12:26.705015 27688 tablet_replica.cc:330] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e: stopping tablet replica
I20260928 13:12:26.705155 27688 raft_consensus.cc:2229] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:12:26.705279 27688 raft_consensus.cc:2258] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 13:12:26.705495 27187 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:39539 (error 108)
I20260928 13:12:26.705842 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 27391
I20260928 13:12:26.706046 27658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:26.705876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:39539 (local address 127.0.0.1:34484)
0928 13:12:26.705970 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:26.706010 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:39539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 13:12:26.706634 27187 catalog_manager.cc:4584] TS 15d94ec465ae469b97141e3ee8c45814 (127.8.88.129:39539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c7e6544e939844d68b061624bed7130d: Network error: Client connection negotiation failed: client connection to 127.8.88.129:39539: connect: Connection refused (error 111)
I20260928 13:12:26.706974 27688 ts_tablet_manager.cc:1905] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:12:26.708616 27688 ts_tablet_manager.cc:1918] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:12:26.708694 27688 log.cc:1196] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/c7e6544e939844d68b061624bed7130d
I20260928 13:12:26.709020 27688 ts_tablet_manager.cc:1939] T c7e6544e939844d68b061624bed7130d P 7b7327667b5a49b69a3af67c0564051e: Deleting consensus metadata
I20260928 13:12:26.709438 27187 catalog_manager.cc:4841] TS 7b7327667b5a49b69a3af67c0564051e (127.8.88.131:46145): tablet c7e6544e939844d68b061624bed7130d (table test-table [id=ff76150053ee4fdab329c9d15b7e7d85]) successfully deleted
I20260928 13:12:26.711688 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 27522
I20260928 13:12:26.716663 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 27169
2026-09-28T13:12:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (735 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:26 dist-test-slave-p1lc krb5kdc[27692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:26 dist-test-slave-p1lc krb5kdc[27692](info): set up 2 sockets
Sep 28 13:12:26 dist-test-slave-p1lc krb5kdc[27692](info): commencing operation
krb5kdc: starting...
W20260928 13:12:27.768744 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:27 dist-test-slave-p1lc krb5kdc[27692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:12:27.897044 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:36907
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:36907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:27.978945 27705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:27.979185 27705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:27.979210 27705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:27.980530 27705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:27.980587 27705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:27.980603 27705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:27.980616 27705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:27.980628 27705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:27.982296 27705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:36907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:36907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:27.982581 27705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:27.982834 27705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:27.988214 27710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:27.988159 27711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:27.988157 27713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:27.988175 27705 server_base.cc:1034] running on GCE node
I20260928 13:12:27.988623 27705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:27.988973 27705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:27.990136 27705 hybrid_clock.cc:648] HybridClock initialized: now 1790601147990110 us; error 43 us; skew 500 ppm
Sep 28 13:12:27 dist-test-slave-p1lc krb5kdc[27692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601147, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:27.992641 27705 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:12:27.993172 27705 webserver.cc:460] Webserver started at http://127.8.88.190:35285/ using document root <none> and password file <none>
I20260928 13:12:27.993417 27705 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:27.993485 27705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:27.993641 27705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:27.994731 27705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "15b731b932594f3dbe3648dfae5ca39b"
format_stamp: "Formatted at 2026-09-28 13:12:27 on dist-test-slave-p1lc"
server_key: "adce590ce62d10c873ef08e720673027"
server_key_iv: "977645c9c3c54690313177b0b932ea35"
server_key_version: "encryptionkey@0"
I20260928 13:12:27.995152 27705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "15b731b932594f3dbe3648dfae5ca39b"
format_stamp: "Formatted at 2026-09-28 13:12:27 on dist-test-slave-p1lc"
server_key: "adce590ce62d10c873ef08e720673027"
server_key_iv: "977645c9c3c54690313177b0b932ea35"
server_key_version: "encryptionkey@0"
I20260928 13:12:27.996554 27705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:27.997437 27720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:27.997615 27705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:12:27.997723 27705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "15b731b932594f3dbe3648dfae5ca39b"
format_stamp: "Formatted at 2026-09-28 13:12:27 on dist-test-slave-p1lc"
server_key: "adce590ce62d10c873ef08e720673027"
server_key_iv: "977645c9c3c54690313177b0b932ea35"
server_key_version: "encryptionkey@0"
I20260928 13:12:27.997812 27705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:28.010615 27705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:28.012660 27705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:28.012838 27705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:28.016671 27705 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:36907
I20260928 13:12:28.016738 27772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:36907 every 8 connection(s)
I20260928 13:12:28.017122 27705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:28.018036 27773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:28.020258 27773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b: Bootstrap starting.
I20260928 13:12:28.020846 27773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:28.021193 27773 log.cc:824] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:28.021938 27773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b: No bootstrap required, opened a new log
I20260928 13:12:28.022356 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 27705
I20260928 13:12:28.022449 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:28.022653 8546 external_mini_cluster.cc:1428] Setting key 87e47326cc073ae259c522cd0a4d1a0d
I20260928 13:12:28.023895 27773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b731b932594f3dbe3648dfae5ca39b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36907 } }
I20260928 13:12:28.024052 27773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:28.024096 27773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b731b932594f3dbe3648dfae5ca39b, State: Initialized, Role: FOLLOWER
I20260928 13:12:28.024217 27773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b731b932594f3dbe3648dfae5ca39b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36907 } }
I20260928 13:12:28.024307 27773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:28.024349 27773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:28.024420 27773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:28.025388 27773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b731b932594f3dbe3648dfae5ca39b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36907 } }
I20260928 13:12:28.025506 27773 leader_election.cc:302] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15b731b932594f3dbe3648dfae5ca39b; no voters:
I20260928 13:12:28.025695 27773 leader_election.cc:288] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:28.025717 27778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:28.025837 27778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [term 1 LEADER]: Becoming Leader. State: Replica: 15b731b932594f3dbe3648dfae5ca39b, State: Running, Role: LEADER
I20260928 13:12:28.025920 27778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b731b932594f3dbe3648dfae5ca39b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36907 } }
I20260928 13:12:28.025966 27773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:28.026247 27779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15b731b932594f3dbe3648dfae5ca39b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b731b932594f3dbe3648dfae5ca39b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36907 } } }
I20260928 13:12:28.026511 27779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:28.026435 27780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15b731b932594f3dbe3648dfae5ca39b. Latest consensus state: current_term: 1 leader_uuid: "15b731b932594f3dbe3648dfae5ca39b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b731b932594f3dbe3648dfae5ca39b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36907 } } }
I20260928 13:12:28.026751 27780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:28.027135 27787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:28.027606 27787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:28.029628 27787 catalog_manager.cc:1347] Generated new cluster ID: 00c7ecfed34a46ea9e27a3d5a687a3a5
I20260928 13:12:28.029703 27787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 13:12:28 dist-test-slave-p1lc krb5kdc[27692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:28.034478 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.023193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42234 (local address 127.8.88.190:36907)
0928 13:12:28.023384 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:28.023394 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:28.023428 (+ 34us) server_negotiation.cc:407] Connection header received
0928 13:12:28.023881 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:28.023889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:28.023941 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:28.024087 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:28.024854 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.025999 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.027127 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.027716 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.028184 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.030797 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:28.030801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:28.030801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:28.030813 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:28.032794 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:28.033214 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:28.033217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:28.033218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:28.033344 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:28.033575 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:28.033576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:28.033577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:28.033882 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:28.034026 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:28.034150 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 13:12:28.034281 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260928 13:12:28.043859 27787 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:28.044394 27787 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:28.055887 27787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b: Generated new TSK 0
I20260928 13:12:28.056154 27787 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.88.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
Entry for principal kudu/127.8.88.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab.
I20260928 13:12:28.093127 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36907
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:28.184587 27799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:28.184818 27799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:28.184859 27799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:28.186034 27799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:28.186105 27799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:28.186151 27799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:12:28.187630 27799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.129.keytab
--principal=kudu/127.8.88.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:28.187896 27799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:28.188144 27799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:28.193656 27804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:28.193619 27807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:28.193634 27805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:28.193979 27799 server_base.cc:1034] running on GCE node
I20260928 13:12:28.194186 27799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:28.194406 27799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:28.195561 27799 hybrid_clock.cc:648] HybridClock initialized: now 1790601148195547 us; error 28 us; skew 500 ppm
Sep 28 13:12:28 dist-test-slave-p1lc krb5kdc[27692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601148, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:28.198348 27799 init.cc:377] Logged in from keytab as kudu/127.8.88.129@KRBTEST.COM (short username kudu)
I20260928 13:12:28.198899 27799 webserver.cc:460] Webserver started at http://127.8.88.129:36365/ using document root <none> and password file <none>
I20260928 13:12:28.199133 27799 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:28.199246 27799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:28.199411 27799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:28.200572 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "0587af8199b9411491e30cb758e27bec"
format_stamp: "Formatted at 2026-09-28 13:12:28 on dist-test-slave-p1lc"
server_key: "455f92aa75a9da91a3af7ac0c27bc2ab"
server_key_iv: "3abc07055f9baf140fb65ab35155975f"
server_key_version: "encryptionkey@0"
I20260928 13:12:28.201052 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "0587af8199b9411491e30cb758e27bec"
format_stamp: "Formatted at 2026-09-28 13:12:28 on dist-test-slave-p1lc"
server_key: "455f92aa75a9da91a3af7ac0c27bc2ab"
server_key_iv: "3abc07055f9baf140fb65ab35155975f"
server_key_version: "encryptionkey@0"
I20260928 13:12:28.202519 27799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:12:28.203378 27814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:28.203547 27799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 13:12:28.203709 27799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "0587af8199b9411491e30cb758e27bec"
format_stamp: "Formatted at 2026-09-28 13:12:28 on dist-test-slave-p1lc"
server_key: "455f92aa75a9da91a3af7ac0c27bc2ab"
server_key_iv: "3abc07055f9baf140fb65ab35155975f"
server_key_version: "encryptionkey@0"
I20260928 13:12:28.203794 27799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:28.219278 27799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:28.221510 27799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:28.221701 27799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:28.221944 27799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:28.222265 27799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:28.222318 27799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:28.222354 27799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:28.222381 27799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:28.228683 27799 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:36941
I20260928 13:12:28.228770 27927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:36941 every 8 connection(s)
I20260928 13:12:28.229214 27799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
Sep 28 13:12:28 dist-test-slave-p1lc krb5kdc[27692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601148, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.129@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:28.238934 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 27799
I20260928 13:12:28.239071 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:12:28.239364 8546 external_mini_cluster.cc:1428] Setting key 6f75b8805f83f0bb898550eae851e881
I20260928 13:12:28.239530 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.229991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:51037 (local address 127.8.88.190:36907)
0928 13:12:28.230064 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:28.230066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:28.230256 (+ 190us) server_negotiation.cc:407] Connection header received
0928 13:12:28.230801 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:28.230803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:28.230819 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:28.230866 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:28.231881 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.232402 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.233541 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.233811 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.233936 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.236275 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:28.236276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:28.236276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:28.236286 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:28.237717 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:28.238163 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:28.238165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:28.238165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:28.238226 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:28.238568 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:28.238569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:28.238570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:28.238755 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:28.238818 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:28.239326 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 13:12:28.239434 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 13:12:28.239738 27930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.229958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:36907 (local address 127.8.88.129:51037)
0928 13:12:28.230124 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:28.230175 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:28.230682 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:28.230937 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:28.230941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:28.231553 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:28.231763 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.231767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.232514 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.232515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.233445 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.233448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.233931 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.233933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.234079 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.234684 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:28.234693 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:28.236164 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:28.237802 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:28.237809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:28.237811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:28.238054 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:28.238333 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:28.238334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:28.238334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:28.238481 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:28.238935 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:28.238943 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:28.239148 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 13:12:28.239276 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 13:12:28.239475 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260928 13:12:28.240286 27928 heartbeater.cc:347] Connected to a master server at 127.8.88.190:36907
I20260928 13:12:28.240402 27928 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:28.240605 27928 heartbeater.cc:511] Master 127.8.88.190:36907 requested a full tablet report, sending...
I20260928 13:12:28.241256 27737 ts_manager.cc:194] Registered new tserver with Master: 0587af8199b9411491e30cb758e27bec (127.8.88.129:36941)
I20260928 13:12:28.242282 27737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.129@KRBTEST.COM'} at 127.8.88.129:51037
WARNING: no policy specified for kudu/127.8.88.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
Entry for principal kudu/127.8.88.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab.
I20260928 13:12:28.268085 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36907
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:28.351902 27933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:28.352134 27933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:28.352159 27933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:28.353490 27933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:28.353538 27933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:28.353566 27933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:12:28.355118 27933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.130.keytab
--principal=kudu/127.8.88.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:28.355381 27933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:28.355620 27933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:28.361027 27941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:28.361156 27933 server_base.cc:1034] running on GCE node
W20260928 13:12:28.361047 27939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:28.361032 27938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:28.361579 27933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:28.361949 27933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:28.363125 27933 hybrid_clock.cc:648] HybridClock initialized: now 1790601148363103 us; error 41 us; skew 500 ppm
Sep 28 13:12:28 dist-test-slave-p1lc krb5kdc[27692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601148, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:28.369582 27933 init.cc:377] Logged in from keytab as kudu/127.8.88.130@KRBTEST.COM (short username kudu)
I20260928 13:12:28.370082 27933 webserver.cc:460] Webserver started at http://127.8.88.130:42951/ using document root <none> and password file <none>
I20260928 13:12:28.370333 27933 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:28.370380 27933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:28.370550 27933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:28.371698 27933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "2dcb968b6a924d428eadafc552f03fce"
format_stamp: "Formatted at 2026-09-28 13:12:28 on dist-test-slave-p1lc"
server_key: "b3a169e12e6af8fff11bab375613b315"
server_key_iv: "df6e5e7b74e6b02daf035cb3404af2b6"
server_key_version: "encryptionkey@0"
I20260928 13:12:28.372064 27933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "2dcb968b6a924d428eadafc552f03fce"
format_stamp: "Formatted at 2026-09-28 13:12:28 on dist-test-slave-p1lc"
server_key: "b3a169e12e6af8fff11bab375613b315"
server_key_iv: "df6e5e7b74e6b02daf035cb3404af2b6"
server_key_version: "encryptionkey@0"
I20260928 13:12:28.373502 27933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:28.374303 27948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:28.374511 27933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:28.374603 27933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "2dcb968b6a924d428eadafc552f03fce"
format_stamp: "Formatted at 2026-09-28 13:12:28 on dist-test-slave-p1lc"
server_key: "b3a169e12e6af8fff11bab375613b315"
server_key_iv: "df6e5e7b74e6b02daf035cb3404af2b6"
server_key_version: "encryptionkey@0"
I20260928 13:12:28.374680 27933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:28.382614 27933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:28.384991 27933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:28.385152 27933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:28.385399 27933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:28.385716 27933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:28.385774 27933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:28.385808 27933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:28.385854 27933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:28.391559 27933 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:39499
I20260928 13:12:28.391625 28061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:39499 every 8 connection(s)
I20260928 13:12:28.391988 27933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:12:28.393414 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 27933
I20260928 13:12:28.393543 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:12:28.393877 8546 external_mini_cluster.cc:1428] Setting key 998b43cb0440d2d5db31811d7c39993f
Sep 28 13:12:28 dist-test-slave-p1lc krb5kdc[27692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601148, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.130@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.88.131@KRBTEST.COM; defaulting to no policy
I20260928 13:12:28.404206 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.392915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:36665 (local address 127.8.88.190:36907)
0928 13:12:28.393004 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:28.393006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:28.393170 (+ 164us) server_negotiation.cc:407] Connection header received
0928 13:12:28.394007 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:28.394008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:28.394022 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:28.394174 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:28.395434 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.396194 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.397943 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.398142 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.398215 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.400946 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:28.400948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:28.400949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:28.400964 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:28.402649 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:28.403204 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:28.403205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:28.403205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:28.403267 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:28.403527 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:28.403528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:28.403528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:28.403649 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:28.403725 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:28.404014 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 13:12:28.404103 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 13:12:28.404400 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.392795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:36907 (local address 127.8.88.130:36665)
0928 13:12:28.393027 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:28.393079 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:28.393853 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:28.394285 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:28.394289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:28.395085 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:28.395318 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.395324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.396305 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.396307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.397827 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.397831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.398247 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.398248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.398411 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.398995 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:28.399007 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:28.400814 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:28.402799 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:28.402814 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:28.402817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:28.403104 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:28.403338 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:28.403339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:28.403340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:28.403454 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:28.403718 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:28.403724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:28.403872 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 13:12:28.403979 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 13:12:28.404149 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":121,"threads_started":1}
I20260928 13:12:28.404956 28062 heartbeater.cc:347] Connected to a master server at 127.8.88.190:36907
I20260928 13:12:28.405073 28062 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:28.405316 28062 heartbeater.cc:511] Master 127.8.88.190:36907 requested a full tablet report, sending...
I20260928 13:12:28.405776 27737 ts_manager.cc:194] Registered new tserver with Master: 2dcb968b6a924d428eadafc552f03fce (127.8.88.130:39499)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.131@KRBTEST.COM" created.
I20260928 13:12:28.406253 27737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.130@KRBTEST.COM'} at 127.8.88.130:36665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
Entry for principal kudu/127.8.88.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab.
I20260928 13:12:28.428601 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36907
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:36859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:28.522085 28067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:28.522322 28067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:28.522351 28067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:28.523703 28067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:28.523762 28067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:28.523793 28067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:12:28.525702 28067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:36859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.131.keytab
--principal=kudu/127.8.88.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:36907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:28.525987 28067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:28.526320 28067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:28.531889 28072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:28.531857 28075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:28.531935 28073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:28.532284 28067 server_base.cc:1034] running on GCE node
I20260928 13:12:28.532557 28067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:28.532800 28067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:28.533974 28067 hybrid_clock.cc:648] HybridClock initialized: now 1790601148533952 us; error 38 us; skew 500 ppm
Sep 28 13:12:28 dist-test-slave-p1lc krb5kdc[27692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601148, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:28.544924 28067 init.cc:377] Logged in from keytab as kudu/127.8.88.131@KRBTEST.COM (short username kudu)
I20260928 13:12:28.545526 28067 webserver.cc:460] Webserver started at http://127.8.88.131:41055/ using document root <none> and password file <none>
I20260928 13:12:28.545809 28067 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:28.545887 28067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:28.546049 28067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:28.547168 28067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "98c2fad45bbd4e66a4f5861f77d797a0"
format_stamp: "Formatted at 2026-09-28 13:12:28 on dist-test-slave-p1lc"
server_key: "b653117889e94753e6d387809799e47c"
server_key_iv: "044a0838cb7315f0ae01533e63b4e078"
server_key_version: "encryptionkey@0"
I20260928 13:12:28.547545 28067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "98c2fad45bbd4e66a4f5861f77d797a0"
format_stamp: "Formatted at 2026-09-28 13:12:28 on dist-test-slave-p1lc"
server_key: "b653117889e94753e6d387809799e47c"
server_key_iv: "044a0838cb7315f0ae01533e63b4e078"
server_key_version: "encryptionkey@0"
I20260928 13:12:28.549000 28067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:28.549806 28082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:28.549998 28067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:28.550099 28067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "98c2fad45bbd4e66a4f5861f77d797a0"
format_stamp: "Formatted at 2026-09-28 13:12:28 on dist-test-slave-p1lc"
server_key: "b653117889e94753e6d387809799e47c"
server_key_iv: "044a0838cb7315f0ae01533e63b4e078"
server_key_version: "encryptionkey@0"
I20260928 13:12:28.550191 28067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:28.569584 28067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:28.571753 28067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:28.571924 28067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:28.572171 28067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:28.572516 28067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:28.572570 28067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:28.572616 28067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:28.572640 28067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:28.578958 28067 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:35647
I20260928 13:12:28.579020 28195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:35647 every 8 connection(s)
I20260928 13:12:28.579362 28067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:12:28.585366 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 28067
I20260928 13:12:28.585497 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:12:28.585691 8546 external_mini_cluster.cc:1428] Setting key 9c793b52a3c36d79ccf9adaabdb3ce56
Sep 28 13:12:28 dist-test-slave-p1lc krb5kdc[27692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601148, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.131@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:28.589416 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.580128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:40807 (local address 127.8.88.190:36907)
0928 13:12:28.580195 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:28.580197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:28.580397 (+ 200us) server_negotiation.cc:407] Connection header received
0928 13:12:28.581004 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:28.581005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:28.581017 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:28.581076 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:28.582026 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.582535 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.583669 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.583887 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.583942 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.586513 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:28.586514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:28.586514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:28.586521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:28.587970 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:28.588402 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:28.588403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:28.588403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:28.588458 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:28.588730 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:28.588731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:28.588731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:28.588862 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:28.588967 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:28.589276 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 13:12:28.589349 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 13:12:28.589682 28198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.580107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:36907 (local address 127.8.88.131:40807)
0928 13:12:28.580264 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:28.580308 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:28.580850 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:28.581133 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:28.581137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:28.581743 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:28.581938 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.581942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.582621 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.582622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.583590 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.583592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.583967 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.583968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.584075 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.584603 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:28.584612 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:28.586426 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:28.588066 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:28.588073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:28.588075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:28.588318 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:28.588544 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:28.588544 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:28.588544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:28.588670 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:28.589024 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:28.589030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:28.589159 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 13:12:28.589248 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 13:12:28.589405 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260928 13:12:28.590361 28196 heartbeater.cc:347] Connected to a master server at 127.8.88.190:36907
I20260928 13:12:28.590452 28196 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:28.590598 28196 heartbeater.cc:511] Master 127.8.88.190:36907 requested a full tablet report, sending...
I20260928 13:12:28.590983 27737 ts_manager.cc:194] Registered new tserver with Master: 98c2fad45bbd4e66a4f5861f77d797a0 (127.8.88.131:35647)
I20260928 13:12:28.591331 27737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.88.131@KRBTEST.COM'} at 127.8.88.131:40807
I20260928 13:12:28.597776 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:12:28.605839 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.599176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.8.88.190:36907)
0928 13:12:28.599240 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:28.599242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:28.599315 (+ 73us) server_negotiation.cc:407] Connection header received
0928 13:12:28.599421 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:28.599421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:28.599432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:28.599500 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:28.600137 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.600689 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.601540 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.601761 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.601822 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.603157 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:28.603159 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:28.603159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:28.603166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:28.604488 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:28.604821 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:28.604822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:28.604822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:28.605127 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:28.605344 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:28.605345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:28.605346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:28.605525 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:28.605618 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:28.605702 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 13:12:28.605771 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 13:12:28.607719 27737 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:12:28.609392 27737 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:12:28.616531 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.611966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50498 (local address 127.8.88.130:39499)
0928 13:12:28.612714 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:28.612719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:28.612728 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:12:28.612768 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:28.612772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:28.612786 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:28.613059 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:28.613134 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.614249 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.615544 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.616221 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.616282 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.616337 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:28.616441 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 13:12:28.616471 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":103,"threads_started":1}
I20260928 13:12:28.616530 28207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.611842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:39499 (local address 127.0.0.1:50498)
0928 13:12:28.611996 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:28.612042 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:28.612128 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:28.612910 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:28.612911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:28.612913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:28.612981 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.612982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.614325 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.614326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.615447 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.615450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.616313 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.616314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.616375 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.616378 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:12:28.616427 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 13:12:28.616459 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260928 13:12:28.616763 28211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.612250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:36941 (local address 127.0.0.1:38818)
0928 13:12:28.612551 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:28.612562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:28.612606 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:28.612921 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:28.612921 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:28.612923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:28.612987 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.612988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.614293 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.614294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.615338 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.615340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.616562 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.616563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.616618 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.616621 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:12:28.616670 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 13:12:28.616705 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":30,"thread_start_us":73,"threads_started":1}
I20260928 13:12:28.616792 28210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.612173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.8.88.131:35647)
0928 13:12:28.612367 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:28.612372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:28.612384 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:12:28.612427 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:28.612430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:28.612446 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:28.612547 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:28.612782 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.613713 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.615296 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.615924 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.616111 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.616188 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:28.616673 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 13:12:28.616716 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260928 13:12:28.616530 28209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.612068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:35647 (local address 127.0.0.1:60090)
0928 13:12:28.612222 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:28.612232 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:28.612285 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:28.612582 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:28.612585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:28.612589 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:28.612667 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.612669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.613830 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.613832 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.615078 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.615080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.616109 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.616110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.616216 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.616231 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 13:12:28.616343 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 13:12:28.616416 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260928 13:12:28.617368 28212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.612363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.8.88.129:36941)
0928 13:12:28.612591 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:28.612597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:28.612608 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:12:28.612729 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:28.612733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:28.612749 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:28.612819 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:28.613062 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.614173 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.615428 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.616210 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.616287 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.616337 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:28.616898 (+ 561us) server_negotiation.cc:299] Negotiation successful
0928 13:12:28.616949 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
I20260928 13:12:28.617623 28130 tablet_service.cc:1467] Processing CreateTablet for tablet 00711582cbc64ceeb73f458d38782f27 (DEFAULT_TABLE table=test-table [id=2ee714fdb5744c94b3539095516b4b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:28.617676 27996 tablet_service.cc:1467] Processing CreateTablet for tablet 00711582cbc64ceeb73f458d38782f27 (DEFAULT_TABLE table=test-table [id=2ee714fdb5744c94b3539095516b4b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:28.617951 28130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00711582cbc64ceeb73f458d38782f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:28.618017 27996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00711582cbc64ceeb73f458d38782f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:28.620633 28213 tablet_bootstrap.cc:492] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce: Bootstrap starting.
I20260928 13:12:28.620684 28214 tablet_bootstrap.cc:492] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0: Bootstrap starting.
I20260928 13:12:28.621251 27862 tablet_service.cc:1467] Processing CreateTablet for tablet 00711582cbc64ceeb73f458d38782f27 (DEFAULT_TABLE table=test-table [id=2ee714fdb5744c94b3539095516b4b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:28.621361 28214 tablet_bootstrap.cc:654] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:28.621559 27862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00711582cbc64ceeb73f458d38782f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:28.621945 28214 log.cc:824] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:28.622623 28214 tablet_bootstrap.cc:492] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0: No bootstrap required, opened a new log
I20260928 13:12:28.622685 28214 ts_tablet_manager.cc:1397] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:12:28.624442 28216 tablet_bootstrap.cc:492] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec: Bootstrap starting.
I20260928 13:12:28.624374 28214 raft_consensus.cc:348] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dcb968b6a924d428eadafc552f03fce" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39499 } } peers { permanent_uuid: "98c2fad45bbd4e66a4f5861f77d797a0" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35647 } } peers { permanent_uuid: "0587af8199b9411491e30cb758e27bec" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 36941 } }
I20260928 13:12:28.624563 28214 raft_consensus.cc:374] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:28.624599 28214 raft_consensus.cc:729] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c2fad45bbd4e66a4f5861f77d797a0, State: Initialized, Role: FOLLOWER
I20260928 13:12:28.624765 28214 consensus_queue.cc:260] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dcb968b6a924d428eadafc552f03fce" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39499 } } peers { permanent_uuid: "98c2fad45bbd4e66a4f5861f77d797a0" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35647 } } peers { permanent_uuid: "0587af8199b9411491e30cb758e27bec" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 36941 } }
I20260928 13:12:28.625097 28214 ts_tablet_manager.cc:1428] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:12:28.625120 28216 tablet_bootstrap.cc:654] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:28.625133 28213 tablet_bootstrap.cc:654] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:28.625223 28196 heartbeater.cc:503] Master 127.8.88.190:36907 was elected leader, sending a full tablet report...
I20260928 13:12:28.625708 28216 log.cc:824] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:28.625735 28213 log.cc:824] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:28.626585 28213 tablet_bootstrap.cc:492] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce: No bootstrap required, opened a new log
I20260928 13:12:28.626587 28216 tablet_bootstrap.cc:492] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec: No bootstrap required, opened a new log
I20260928 13:12:28.626655 28216 ts_tablet_manager.cc:1397] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 13:12:28.626658 28213 ts_tablet_manager.cc:1397] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260928 13:12:28.628290 28213 raft_consensus.cc:348] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dcb968b6a924d428eadafc552f03fce" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39499 } } peers { permanent_uuid: "98c2fad45bbd4e66a4f5861f77d797a0" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35647 } } peers { permanent_uuid: "0587af8199b9411491e30cb758e27bec" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 36941 } }
I20260928 13:12:28.628448 28213 raft_consensus.cc:374] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:28.628373 28216 raft_consensus.cc:348] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dcb968b6a924d428eadafc552f03fce" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39499 } } peers { permanent_uuid: "98c2fad45bbd4e66a4f5861f77d797a0" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35647 } } peers { permanent_uuid: "0587af8199b9411491e30cb758e27bec" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 36941 } }
I20260928 13:12:28.628492 28213 raft_consensus.cc:729] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dcb968b6a924d428eadafc552f03fce, State: Initialized, Role: FOLLOWER
I20260928 13:12:28.628535 28216 raft_consensus.cc:374] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:28.628579 28216 raft_consensus.cc:729] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0587af8199b9411491e30cb758e27bec, State: Initialized, Role: FOLLOWER
I20260928 13:12:28.628643 28213 consensus_queue.cc:260] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dcb968b6a924d428eadafc552f03fce" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39499 } } peers { permanent_uuid: "98c2fad45bbd4e66a4f5861f77d797a0" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35647 } } peers { permanent_uuid: "0587af8199b9411491e30cb758e27bec" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 36941 } }
I20260928 13:12:28.628686 28216 consensus_queue.cc:260] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dcb968b6a924d428eadafc552f03fce" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39499 } } peers { permanent_uuid: "98c2fad45bbd4e66a4f5861f77d797a0" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35647 } } peers { permanent_uuid: "0587af8199b9411491e30cb758e27bec" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 36941 } }
I20260928 13:12:28.628960 28062 heartbeater.cc:503] Master 127.8.88.190:36907 was elected leader, sending a full tablet report...
I20260928 13:12:28.628996 28216 ts_tablet_manager.cc:1428] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 13:12:28.628968 28213 ts_tablet_manager.cc:1428] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 13:12:28.629115 27928 heartbeater.cc:503] Master 127.8.88.190:36907 was elected leader, sending a full tablet report...
W20260928 13:12:28.642669 28063 tablet.cc:2369] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 13:12:28.729990 27929 tablet.cc:2369] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 13:12:28.756121 28220 raft_consensus.cc:482] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:12:28.756251 28220 raft_consensus.cc:504] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dcb968b6a924d428eadafc552f03fce" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39499 } } peers { permanent_uuid: "98c2fad45bbd4e66a4f5861f77d797a0" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35647 } } peers { permanent_uuid: "0587af8199b9411491e30cb758e27bec" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 36941 } }
I20260928 13:12:28.756593 28220 leader_election.cc:288] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98c2fad45bbd4e66a4f5861f77d797a0 (127.8.88.131:35647), 0587af8199b9411491e30cb758e27bec (127.8.88.129:36941)
I20260928 13:12:28.759599 28064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.756806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:35647 (local address 127.8.88.130:56913)
0928 13:12:28.756934 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:28.756948 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:28.757038 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:28.757209 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:28.757211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:28.757213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:28.757294 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.757296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.757975 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.757976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.758729 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.758731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.759318 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.759319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.759402 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.759405 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:12:28.759482 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 13:12:28.759525 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 13:12:28.759599 28210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.756924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:56913 (local address 127.8.88.131:35647)
0928 13:12:28.756975 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:28.756976 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:28.757097 (+ 121us) server_negotiation.cc:407] Connection header received
0928 13:12:28.757129 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:28.757130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:28.757140 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:28.757190 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:28.757397 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.757882 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.758831 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.759218 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.759318 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.759328 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:28.759490 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 13:12:28.759528 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 13:12:28.759964 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00711582cbc64ceeb73f458d38782f27" candidate_uuid: "2dcb968b6a924d428eadafc552f03fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c2fad45bbd4e66a4f5861f77d797a0" is_pre_election: true
I20260928 13:12:28.760165 28150 raft_consensus.cc:2454] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dcb968b6a924d428eadafc552f03fce in term 0.
I20260928 13:12:28.760228 28212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.757077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:55769 (local address 127.8.88.129:36941)
0928 13:12:28.757120 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:28.757122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:28.757252 (+ 130us) server_negotiation.cc:407] Connection header received
0928 13:12:28.757377 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:28.757378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:28.757389 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:28.757421 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:28.757617 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.758076 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.759265 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.759853 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.759939 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.759948 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:28.760120 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 13:12:28.760159 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 13:12:28.760268 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.757003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:36941 (local address 127.8.88.130:55769)
0928 13:12:28.757195 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:28.757205 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:28.757296 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:28.757474 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:28.757474 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:28.757475 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:28.757536 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.757537 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.758163 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.758164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.759118 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:28.759121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.759946 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.759947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:28.760027 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.760036 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 13:12:28.760109 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 13:12:28.760159 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":113,"threads_started":1}
I20260928 13:12:28.760421 27951 leader_election.cc:302] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2dcb968b6a924d428eadafc552f03fce, 98c2fad45bbd4e66a4f5861f77d797a0; no voters:
I20260928 13:12:28.760592 28220 raft_consensus.cc:2781] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:12:28.760670 28220 raft_consensus.cc:482] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:12:28.760704 28220 raft_consensus.cc:3037] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:28.760774 27882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00711582cbc64ceeb73f458d38782f27" candidate_uuid: "2dcb968b6a924d428eadafc552f03fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0587af8199b9411491e30cb758e27bec" is_pre_election: true
I20260928 13:12:28.760962 27882 raft_consensus.cc:2454] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2dcb968b6a924d428eadafc552f03fce in term 0.
I20260928 13:12:28.761533 28220 raft_consensus.cc:504] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dcb968b6a924d428eadafc552f03fce" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39499 } } peers { permanent_uuid: "98c2fad45bbd4e66a4f5861f77d797a0" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35647 } } peers { permanent_uuid: "0587af8199b9411491e30cb758e27bec" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 36941 } }
I20260928 13:12:28.761689 28220 leader_election.cc:288] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [CANDIDATE]: Term 1 election: Requested vote from peers 98c2fad45bbd4e66a4f5861f77d797a0 (127.8.88.131:35647), 0587af8199b9411491e30cb758e27bec (127.8.88.129:36941)
I20260928 13:12:28.761947 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00711582cbc64ceeb73f458d38782f27" candidate_uuid: "2dcb968b6a924d428eadafc552f03fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c2fad45bbd4e66a4f5861f77d797a0"
I20260928 13:12:28.761950 27882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00711582cbc64ceeb73f458d38782f27" candidate_uuid: "2dcb968b6a924d428eadafc552f03fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0587af8199b9411491e30cb758e27bec"
I20260928 13:12:28.762024 27882 raft_consensus.cc:3037] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:28.762044 28150 raft_consensus.cc:3037] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:28.762737 28150 raft_consensus.cc:2454] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dcb968b6a924d428eadafc552f03fce in term 1.
I20260928 13:12:28.762737 27882 raft_consensus.cc:2454] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2dcb968b6a924d428eadafc552f03fce in term 1.
I20260928 13:12:28.762928 27951 leader_election.cc:302] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0587af8199b9411491e30cb758e27bec, 2dcb968b6a924d428eadafc552f03fce; no voters:
I20260928 13:12:28.763043 28220 raft_consensus.cc:2781] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:28.763216 28220 raft_consensus.cc:686] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [term 1 LEADER]: Becoming Leader. State: Replica: 2dcb968b6a924d428eadafc552f03fce, State: Running, Role: LEADER
I20260928 13:12:28.763317 28220 consensus_queue.cc:237] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dcb968b6a924d428eadafc552f03fce" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39499 } } peers { permanent_uuid: "98c2fad45bbd4e66a4f5861f77d797a0" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35647 } } peers { permanent_uuid: "0587af8199b9411491e30cb758e27bec" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 36941 } }
I20260928 13:12:28.764155 27736 catalog_manager.cc:5495] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dcb968b6a924d428eadafc552f03fce (127.8.88.130). New cstate: current_term: 1 leader_uuid: "2dcb968b6a924d428eadafc552f03fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2dcb968b6a924d428eadafc552f03fce" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39499 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98c2fad45bbd4e66a4f5861f77d797a0" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0587af8199b9411491e30cb758e27bec" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 36941 } health_report { overall_health: UNKNOWN } } }
I20260928 13:12:28.798647 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.796089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.8.88.130:39499)
0928 13:12:28.796194 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:28.796196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:28.796204 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:12:28.796241 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:28.796241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:28.796260 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:28.796328 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:28.796470 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.797037 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.797759 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.797965 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.798019 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.798210 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:28.798318 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:28.798394 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:28.798512 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 13:12:28.798532 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 13:12:28.800845 28150 raft_consensus.cc:1261] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0 [term 1 FOLLOWER]: Refusing update from remote peer 2dcb968b6a924d428eadafc552f03fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:28.801054 27882 raft_consensus.cc:1261] T 00711582cbc64ceeb73f458d38782f27 P 0587af8199b9411491e30cb758e27bec [term 1 FOLLOWER]: Refusing update from remote peer 2dcb968b6a924d428eadafc552f03fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:28.801370 28220 consensus_queue.cc:1035] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "98c2fad45bbd4e66a4f5861f77d797a0" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 35647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:28.801502 28223 consensus_queue.cc:1035] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "0587af8199b9411491e30cb758e27bec" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 36941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:28.804898 28228 mvcc.cc:205] Tried to move back new op lower bound from 7334302305484779520 to 7334302305335001088. Current Snapshot: MvccSnapshot[applied={T|T < 7334302305484779520 or (T in {7334302305484779520})}]
I20260928 13:12:28.806620 27737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:12:28.806728 27737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:12:28.807612 27737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15b731b932594f3dbe3648dfae5ca39b: Sending DeleteTablet for 3 replicas of tablet 00711582cbc64ceeb73f458d38782f27
I20260928 13:12:28.808100 27996 tablet_service.cc:1514] Processing DeleteTablet for tablet 00711582cbc64ceeb73f458d38782f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:28 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:50498
I20260928 13:12:28.808089 28130 tablet_service.cc:1514] Processing DeleteTablet for tablet 00711582cbc64ceeb73f458d38782f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:28 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:60090
I20260928 13:12:28.808243 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 27799
I20260928 13:12:28.808352 27862 tablet_service.cc:1514] Processing DeleteTablet for tablet 00711582cbc64ceeb73f458d38782f27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:28 UTC) from {username='kudu', principal='kudu/127.8.88.190@KRBTEST.COM'} at 127.0.0.1:38818
I20260928 13:12:28.809039 28236 tablet_replica.cc:330] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0: stopping tablet replica
I20260928 13:12:28.809151 28236 raft_consensus.cc:2229] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:12:28.809266 28236 raft_consensus.cc:2258] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:12:28.809342 28237 tablet_replica.cc:330] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce: stopping tablet replica
I20260928 13:12:28.809552 28237 raft_consensus.cc:2229] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:12:28.809715 28237 raft_consensus.cc:2258] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:12:28.809825 28236 ts_tablet_manager.cc:1905] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:12:28.810228 28237 ts_tablet_manager.cc:1905] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:12:28.811686 28236 ts_tablet_manager.cc:1918] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:12:28.811842 28236 log.cc:1196] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/00711582cbc64ceeb73f458d38782f27
I20260928 13:12:28.811950 28237 ts_tablet_manager.cc:1918] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:12:28.812026 28237 log.cc:1196] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/00711582cbc64ceeb73f458d38782f27
I20260928 13:12:28.812307 28237 ts_tablet_manager.cc:1939] T 00711582cbc64ceeb73f458d38782f27 P 2dcb968b6a924d428eadafc552f03fce: Deleting consensus metadata
I20260928 13:12:28.812309 28236 ts_tablet_manager.cc:1939] T 00711582cbc64ceeb73f458d38782f27 P 98c2fad45bbd4e66a4f5861f77d797a0: Deleting consensus metadata
I20260928 13:12:28.812729 27723 catalog_manager.cc:4841] TS 2dcb968b6a924d428eadafc552f03fce (127.8.88.130:39499): tablet 00711582cbc64ceeb73f458d38782f27 (table test-table [id=2ee714fdb5744c94b3539095516b4b07]) successfully deleted
I20260928 13:12:28.812841 27723 catalog_manager.cc:4841] TS 98c2fad45bbd4e66a4f5861f77d797a0 (127.8.88.131:35647): tablet 00711582cbc64ceeb73f458d38782f27 (table test-table [id=2ee714fdb5744c94b3539095516b4b07]) successfully deleted
W20260928 13:12:28.815191 27723 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:36941 (error 108)
I20260928 13:12:28.815349 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 27933
I20260928 13:12:28.817755 28209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.817609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:36941 (local address 127.0.0.1:38834)
0928 13:12:28.817674 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:28.817722 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:36941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 13:12:28.817937 27723 catalog_manager.cc:4584] TS 0587af8199b9411491e30cb758e27bec (127.8.88.129:36941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00711582cbc64ceeb73f458d38782f27: Network error: Client connection negotiation failed: client connection to 127.8.88.129:36941: connect: Connection refused (error 111)
I20260928 13:12:28.821302 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 28067
I20260928 13:12:28.826048 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 27705
2026-09-28T13:12:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2110 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T13:12:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:28Z Disabled control of system clock
I20260928 13:12:28.854221 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:39061
--webserver_interface=127.8.88.190
--webserver_port=0
--builtin_ntp_servers=127.8.88.148:40927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:39061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:12:28.940698 28242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:28.941030 28242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:28.941056 28242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:28.942241 28242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:28.942282 28242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:28.942296 28242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:28.942308 28242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:28.942320 28242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:28.943667 28242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:39061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:39061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:28.943879 28242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:28.944125 28242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:28.949524 28247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:28.949524 28250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:28.949795 28248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:28.950059 28242 server_base.cc:1034] running on GCE node
I20260928 13:12:28.950344 28242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:28.950665 28242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:28.951881 28242 hybrid_clock.cc:648] HybridClock initialized: now 1790601148951846 us; error 47 us; skew 500 ppm
I20260928 13:12:28.953521 28242 webserver.cc:460] Webserver started at http://127.8.88.190:45409/ using document root <none> and password file <none>
I20260928 13:12:28.953867 28242 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:28.953920 28242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:28.954109 28242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:28.955202 28242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "0ace4024e8964789a19156ce8110a3bb"
format_stamp: "Formatted at 2026-09-28 13:12:28 on dist-test-slave-p1lc"
server_key: "ce9d1c89f7d1c8fa26d8a6d28f711b45"
server_key_iv: "b38fbfd54fe79b4ffec7f58fc094058a"
server_key_version: "encryptionkey@0"
I20260928 13:12:28.955574 28242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "0ace4024e8964789a19156ce8110a3bb"
format_stamp: "Formatted at 2026-09-28 13:12:28 on dist-test-slave-p1lc"
server_key: "ce9d1c89f7d1c8fa26d8a6d28f711b45"
server_key_iv: "b38fbfd54fe79b4ffec7f58fc094058a"
server_key_version: "encryptionkey@0"
I20260928 13:12:28.957042 28242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:28.957937 28256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:28.958171 28242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:28.958249 28242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "0ace4024e8964789a19156ce8110a3bb"
format_stamp: "Formatted at 2026-09-28 13:12:28 on dist-test-slave-p1lc"
server_key: "ce9d1c89f7d1c8fa26d8a6d28f711b45"
server_key_iv: "b38fbfd54fe79b4ffec7f58fc094058a"
server_key_version: "encryptionkey@0"
I20260928 13:12:28.958354 28242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:28.972239 28242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:28.972600 28242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:28.972764 28242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:28.976727 28242 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:39061
I20260928 13:12:28.976778 28308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:39061 every 8 connection(s)
I20260928 13:12:28.977162 28242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:28.978341 28309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:28.978796 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 28242
I20260928 13:12:28.978911 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:28.979103 8546 external_mini_cluster.cc:1428] Setting key e4b736a3ddfbe2d00cf28cf8a55b316f
I20260928 13:12:28.981271 28309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb: Bootstrap starting.
I20260928 13:12:28.981914 28309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:28.982203 28309 log.cc:824] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:28.982931 28309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb: No bootstrap required, opened a new log
I20260928 13:12:28.984733 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:28.979742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51168 (local address 127.8.88.190:39061)
0928 13:12:28.979970 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:28.979978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:28.980003 (+ 25us) server_negotiation.cc:407] Connection header received
0928 13:12:28.980424 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:28.980430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:28.980491 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:28.980645 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:28.980900 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.981852 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.982753 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:28.983231 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:28.983369 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:28.984115 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:28.984120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:28.984120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:28.984295 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:28.984327 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:28.984339 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:28.984407 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:28.984483 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 13:12:28.984565 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260928 13:12:28.985121 28309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ace4024e8964789a19156ce8110a3bb" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39061 } }
I20260928 13:12:28.985286 28309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:28.985337 28309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ace4024e8964789a19156ce8110a3bb, State: Initialized, Role: FOLLOWER
I20260928 13:12:28.985492 28309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ace4024e8964789a19156ce8110a3bb" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39061 } }
I20260928 13:12:28.985594 28309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:28.985638 28309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:28.985687 28309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:28.986263 28309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ace4024e8964789a19156ce8110a3bb" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39061 } }
I20260928 13:12:28.986385 28309 leader_election.cc:302] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ace4024e8964789a19156ce8110a3bb; no voters:
I20260928 13:12:28.986577 28309 leader_election.cc:288] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:28.986618 28314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:28.986850 28309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:28.986874 28314 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [term 1 LEADER]: Becoming Leader. State: Replica: 0ace4024e8964789a19156ce8110a3bb, State: Running, Role: LEADER
I20260928 13:12:28.987015 28314 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ace4024e8964789a19156ce8110a3bb" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39061 } }
I20260928 13:12:28.987448 28315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ace4024e8964789a19156ce8110a3bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ace4024e8964789a19156ce8110a3bb" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39061 } } }
I20260928 13:12:28.987582 28315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:28.987659 28316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ace4024e8964789a19156ce8110a3bb. Latest consensus state: current_term: 1 leader_uuid: "0ace4024e8964789a19156ce8110a3bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ace4024e8964789a19156ce8110a3bb" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39061 } } }
I20260928 13:12:28.987733 28316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:28.987954 28322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:28.988516 28322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:28.990041 28322 catalog_manager.cc:1347] Generated new cluster ID: 2bc35998c85b4c21b70dd045fe937cd5
I20260928 13:12:28.990123 28322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:29.010107 28322 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:29.010715 28322 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:29.017525 28322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb: Generated new TSK 0
I20260928 13:12:29.017763 28322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:29.021046 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.129:0
--local_ip_for_outbound_sockets=127.8.88.129
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:39061
--builtin_ntp_servers=127.8.88.148:40927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:12:29.105366 28333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:29.105643 28333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:29.105672 28333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:29.106930 28333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:29.106973 28333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:29.107000 28333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.129
I20260928 13:12:29.108385 28333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.88.129
--webserver_port=0
--tserver_master_addrs=127.8.88.190:39061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.129
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:29.108657 28333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:29.109021 28333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:29.115151 28341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:29.115154 28339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:29.115214 28338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:29.115175 28333 server_base.cc:1034] running on GCE node
I20260928 13:12:29.115588 28333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:29.115970 28333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:29.117154 28333 hybrid_clock.cc:648] HybridClock initialized: now 1790601149117142 us; error 29 us; skew 500 ppm
I20260928 13:12:29.118497 28333 webserver.cc:460] Webserver started at http://127.8.88.129:43579/ using document root <none> and password file <none>
I20260928 13:12:29.118827 28333 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:29.118912 28333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:29.119063 28333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:29.120165 28333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/data/instance:
uuid: "0c4ae81755704c9c92b6501b65bbc9bb"
format_stamp: "Formatted at 2026-09-28 13:12:29 on dist-test-slave-p1lc"
server_key: "17875c9c55f47a04e7af46a9575acd61"
server_key_iv: "f5473897ad814266a69801bcb00481f4"
server_key_version: "encryptionkey@0"
I20260928 13:12:29.120538 28333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance:
uuid: "0c4ae81755704c9c92b6501b65bbc9bb"
format_stamp: "Formatted at 2026-09-28 13:12:29 on dist-test-slave-p1lc"
server_key: "17875c9c55f47a04e7af46a9575acd61"
server_key_iv: "f5473897ad814266a69801bcb00481f4"
server_key_version: "encryptionkey@0"
I20260928 13:12:29.122081 28333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:29.122900 28347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:29.123122 28333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:29.123229 28333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/wal
uuid: "0c4ae81755704c9c92b6501b65bbc9bb"
format_stamp: "Formatted at 2026-09-28 13:12:29 on dist-test-slave-p1lc"
server_key: "17875c9c55f47a04e7af46a9575acd61"
server_key_iv: "f5473897ad814266a69801bcb00481f4"
server_key_version: "encryptionkey@0"
I20260928 13:12:29.123309 28333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:29.145510 28333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:29.145857 28333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:29.146013 28333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:29.146289 28333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:29.146626 28333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:29.146750 28333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:29.146802 28333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:29.146833 28333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:29.152791 28333 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.129:33577
I20260928 13:12:29.152833 28460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.129:33577 every 8 connection(s)
I20260928 13:12:29.153228 28333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/data/info.pb
I20260928 13:12:29.156358 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 28333
I20260928 13:12:29.156448 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-0/wal/instance
I20260928 13:12:29.156672 8546 external_mini_cluster.cc:1428] Setting key 3dad76b67fde502ecd856c837d70e74b
I20260928 13:12:29.157774 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.130:0
--local_ip_for_outbound_sockets=127.8.88.130
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:39061
--builtin_ntp_servers=127.8.88.148:40927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 13:12:29.158842 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.153982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.129:39425 (local address 127.8.88.190:39061)
0928 13:12:29.154052 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:29.154054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:29.154288 (+ 234us) server_negotiation.cc:407] Connection header received
0928 13:12:29.154759 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:29.154760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:29.154774 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:29.154823 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:29.155464 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.155991 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.157204 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.157488 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.157558 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.158469 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:29.158471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:29.158471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:29.158476 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:29.158492 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:29.158496 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:29.158555 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:29.158618 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 13:12:29.158651 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 13:12:29.158917 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.153987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39061 (local address 127.8.88.129:39425)
0928 13:12:29.154171 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:29.154205 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:29.154678 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:29.154887 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:29.154890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:29.154899 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:29.155389 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.155393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.156076 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.156077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.157130 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.157133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.157579 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.157580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.157689 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.158333 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:12:29.158334 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:29.158362 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:12:29.158385 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:29.158562 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:29.158566 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:29.158568 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 13:12:29.158619 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 13:12:29.158699 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260928 13:12:29.159575 28461 heartbeater.cc:347] Connected to a master server at 127.8.88.190:39061
I20260928 13:12:29.159677 28461 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:29.159886 28461 heartbeater.cc:511] Master 127.8.88.190:39061 requested a full tablet report, sending...
I20260928 13:12:29.160418 28273 ts_manager.cc:194] Registered new tserver with Master: 0c4ae81755704c9c92b6501b65bbc9bb (127.8.88.129:33577)
I20260928 13:12:29.161273 28273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.88.129:39425
W20260928 13:12:29.240928 28464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:29.241160 28464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:29.241184 28464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:29.242439 28464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:29.242484 28464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:29.242509 28464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.130
I20260928 13:12:29.243988 28464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.88.130
--webserver_port=0
--tserver_master_addrs=127.8.88.190:39061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.130
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:29.244254 28464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:29.244565 28464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:29.249878 28472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:29.249889 28469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:29.249900 28470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:29.250188 28464 server_base.cc:1034] running on GCE node
I20260928 13:12:29.250536 28464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:29.250829 28464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:29.251996 28464 hybrid_clock.cc:648] HybridClock initialized: now 1790601149251971 us; error 36 us; skew 500 ppm
I20260928 13:12:29.253491 28464 webserver.cc:460] Webserver started at http://127.8.88.130:36811/ using document root <none> and password file <none>
I20260928 13:12:29.253789 28464 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:29.253840 28464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:29.253993 28464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:29.255072 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/data/instance:
uuid: "a44216177276449ea0220332cb9ce0bc"
format_stamp: "Formatted at 2026-09-28 13:12:29 on dist-test-slave-p1lc"
server_key: "117df9bb0f4d8cc88ca753b17a9b3e40"
server_key_iv: "49baba65a0cbec3f17927dd361a0e02f"
server_key_version: "encryptionkey@0"
I20260928 13:12:29.255434 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance:
uuid: "a44216177276449ea0220332cb9ce0bc"
format_stamp: "Formatted at 2026-09-28 13:12:29 on dist-test-slave-p1lc"
server_key: "117df9bb0f4d8cc88ca753b17a9b3e40"
server_key_iv: "49baba65a0cbec3f17927dd361a0e02f"
server_key_version: "encryptionkey@0"
I20260928 13:12:29.256979 28464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260928 13:12:29.257875 28478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:29.258066 28464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:29.258174 28464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/wal
uuid: "a44216177276449ea0220332cb9ce0bc"
format_stamp: "Formatted at 2026-09-28 13:12:29 on dist-test-slave-p1lc"
server_key: "117df9bb0f4d8cc88ca753b17a9b3e40"
server_key_iv: "49baba65a0cbec3f17927dd361a0e02f"
server_key_version: "encryptionkey@0"
I20260928 13:12:29.258265 28464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:29.267660 28464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:29.267988 28464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:29.268146 28464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:29.268388 28464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:29.268728 28464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:29.268784 28464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:29.268826 28464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:29.268850 28464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:29.274816 28464 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.130:39791
I20260928 13:12:29.274859 28591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.130:39791 every 8 connection(s)
I20260928 13:12:29.275202 28464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/data/info.pb
I20260928 13:12:29.280500 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.275923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.130:45765 (local address 127.8.88.190:39061)
0928 13:12:29.275987 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:29.275991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:29.276272 (+ 281us) server_negotiation.cc:407] Connection header received
0928 13:12:29.276735 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:29.276736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:29.276747 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:29.276812 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:29.277530 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.278093 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.279078 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.279276 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.279334 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.280250 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:29.280251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:29.280251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:29.280255 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:29.280269 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:29.280273 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:29.280328 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:29.280411 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 13:12:29.280440 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 13:12:29.280728 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.275919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39061 (local address 127.8.88.130:45765)
0928 13:12:29.276111 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:29.276153 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:29.276646 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:29.276932 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:29.276937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:29.276947 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:29.277449 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.277453 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.278187 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.278187 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.278994 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.278996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.279353 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.279354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.279462 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.280099 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:12:29.280100 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:29.280128 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:12:29.280151 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:29.280340 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:29.280344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:29.280346 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 13:12:29.280399 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 13:12:29.280479 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260928 13:12:29.281350 28592 heartbeater.cc:347] Connected to a master server at 127.8.88.190:39061
I20260928 13:12:29.281445 28592 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:29.281632 28592 heartbeater.cc:511] Master 127.8.88.190:39061 requested a full tablet report, sending...
I20260928 13:12:29.282017 28273 ts_manager.cc:194] Registered new tserver with Master: a44216177276449ea0220332cb9ce0bc (127.8.88.130:39791)
I20260928 13:12:29.282378 28273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.88.130:45765
I20260928 13:12:29.283599 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 28464
I20260928 13:12:29.283716 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/wal/instance
I20260928 13:12:29.283921 8546 external_mini_cluster.cc:1428] Setting key 3b57d3912567a6e2a68d799b50b1146a
I20260928 13:12:29.285002 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.88.131:0
--local_ip_for_outbound_sockets=127.8.88.131
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:39061
--builtin_ntp_servers=127.8.88.148:40927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 13:12:29.369114 28595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:29.369354 28595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:29.369388 28595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:29.370548 28595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:29.370613 28595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:29.370667 28595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.88.131
I20260928 13:12:29.372080 28595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.88.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.88.131
--webserver_port=0
--tserver_master_addrs=127.8.88.190:39061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.88.131
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:29.372365 28595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:29.372689 28595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:29.378260 28600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:29.378194 28603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:29.378237 28595 server_base.cc:1034] running on GCE node
W20260928 13:12:29.378204 28601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:29.378662 28595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:29.378996 28595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:29.380185 28595 hybrid_clock.cc:648] HybridClock initialized: now 1790601149380172 us; error 33 us; skew 500 ppm
I20260928 13:12:29.381752 28595 webserver.cc:460] Webserver started at http://127.8.88.131:33039/ using document root <none> and password file <none>
I20260928 13:12:29.382081 28595 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:29.382158 28595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:29.382335 28595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:29.383569 28595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/data/instance:
uuid: "9702ad6b01254102af5a1ae47f742b56"
format_stamp: "Formatted at 2026-09-28 13:12:29 on dist-test-slave-p1lc"
server_key: "29c0511ef3831fc1fe3cd0a66994ce87"
server_key_iv: "7fb7b8c781fe16a20b2b137a0486c704"
server_key_version: "encryptionkey@0"
I20260928 13:12:29.384122 28595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance:
uuid: "9702ad6b01254102af5a1ae47f742b56"
format_stamp: "Formatted at 2026-09-28 13:12:29 on dist-test-slave-p1lc"
server_key: "29c0511ef3831fc1fe3cd0a66994ce87"
server_key_iv: "7fb7b8c781fe16a20b2b137a0486c704"
server_key_version: "encryptionkey@0"
I20260928 13:12:29.385854 28595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260928 13:12:29.386768 28609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:29.387046 28595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:12:29.387185 28595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/wal
uuid: "9702ad6b01254102af5a1ae47f742b56"
format_stamp: "Formatted at 2026-09-28 13:12:29 on dist-test-slave-p1lc"
server_key: "29c0511ef3831fc1fe3cd0a66994ce87"
server_key_iv: "7fb7b8c781fe16a20b2b137a0486c704"
server_key_version: "encryptionkey@0"
I20260928 13:12:29.387305 28595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:29.404340 28595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:29.404685 28595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:29.404841 28595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:29.405189 28595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 13:12:29.405525 28595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 13:12:29.405578 28595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:29.405622 28595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 13:12:29.405660 28595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:29.411692 28595 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.131:32961
I20260928 13:12:29.411757 28722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.131:32961 every 8 connection(s)
I20260928 13:12:29.412115 28595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/data/info.pb
I20260928 13:12:29.418089 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.413038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:49089 (local address 127.8.88.190:39061)
0928 13:12:29.413098 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:29.413100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:29.413372 (+ 272us) server_negotiation.cc:407] Connection header received
0928 13:12:29.413870 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:29.413871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:29.413882 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:29.413949 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:29.414832 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.415341 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.416504 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.416721 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.416793 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.417825 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:29.417827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:29.417827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:29.417834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:29.417849 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:29.417853 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:29.417912 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:29.417999 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 13:12:29.418026 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 13:12:29.418360 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.412942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:39061 (local address 127.8.88.131:49089)
0928 13:12:29.413232 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:29.413277 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:29.413757 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:29.414062 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:29.414066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:29.414075 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:29.414681 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.414686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.415476 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.415477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.416383 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.416386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.416814 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.416814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.417031 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.417656 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 13:12:29.417656 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:29.417688 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 13:12:29.417712 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:29.417923 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:29.417927 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:29.417930 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 13:12:29.417988 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 13:12:29.418079 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":94,"threads_started":1}
I20260928 13:12:29.419008 28723 heartbeater.cc:347] Connected to a master server at 127.8.88.190:39061
I20260928 13:12:29.419116 28723 heartbeater.cc:464] Registering TS with master...
I20260928 13:12:29.419317 28723 heartbeater.cc:511] Master 127.8.88.190:39061 requested a full tablet report, sending...
I20260928 13:12:29.419745 28273 ts_manager.cc:194] Registered new tserver with Master: 9702ad6b01254102af5a1ae47f742b56 (127.8.88.131:32961)
I20260928 13:12:29.420169 28273 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.88.131:49089
I20260928 13:12:29.420574 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 28595
I20260928 13:12:29.420689 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/wal/instance
I20260928 13:12:29.420933 8546 external_mini_cluster.cc:1428] Setting key 03ea7b34d9a935ebd416fa8c43bee4ad
I20260928 13:12:29.421725 8546 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 13:12:29.426288 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.423118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60804 (local address 127.8.88.190:39061)
0928 13:12:29.423203 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:29.423205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:29.423293 (+ 88us) server_negotiation.cc:407] Connection header received
0928 13:12:29.423363 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:29.423364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:29.423374 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:29.423424 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:29.423599 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.424122 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.424903 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.425118 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.425196 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.426040 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:29.426041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:29.426042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:29.426047 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:29.426062 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:29.426066 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:29.426113 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:29.426176 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 13:12:29.426199 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 13:12:29.428191 28273 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 13:12:29.430006 28273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 13:12:29.437018 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.432333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:39791 (local address 127.0.0.1:51222)
0928 13:12:29.432769 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:29.432779 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:29.432842 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:29.433024 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:29.433027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:29.433029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:29.433100 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.433103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.434376 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.434377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.435785 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.435788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.436696 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.436697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.436778 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.436792 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 13:12:29.436884 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 13:12:29.436939 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":115,"threads_started":1}
I20260928 13:12:29.437029 28735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.432589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.8.88.130:39791)
0928 13:12:29.432740 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:29.432745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:29.432853 (+ 108us) server_negotiation.cc:407] Connection header received
0928 13:12:29.432933 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:29.432936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:29.432951 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:29.433023 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:29.433202 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.434277 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.435880 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.436591 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.436685 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.436784 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:29.436899 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 13:12:29.436938 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260928 13:12:29.437119 28734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.432539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:33577 (local address 127.0.0.1:49480)
0928 13:12:29.432971 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:29.432981 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:29.433025 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:29.433550 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:29.433551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:29.433552 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:29.433581 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.433582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.434846 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.434847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.435930 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.435932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.436858 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.436859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.436944 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.436948 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:12:29.436999 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 13:12:29.437043 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":72,"threads_started":1}
I20260928 13:12:29.437327 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.432750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.8.88.129:33577)
0928 13:12:29.433336 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:29.433340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:29.433350 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:12:29.433390 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:29.433393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:29.433408 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:29.433478 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:29.433660 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.434716 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.436021 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.436673 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.436741 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.436823 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:29.437201 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 13:12:29.437241 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":104,"threads_started":1}
I20260928 13:12:29.438067 28732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.432305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.131:32961 (local address 127.0.0.1:34078)
0928 13:12:29.432507 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:29.432549 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:29.432625 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:29.433055 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:29.433056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:29.433058 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:29.433112 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.433114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.435396 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.435397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.436300 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.436302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.437811 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.437822 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.437888 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.437893 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 13:12:29.437953 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 13:12:29.437998 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":78,"threads_started":1}
I20260928 13:12:29.437959 28395 tablet_service.cc:1467] Processing CreateTablet for tablet 141355250c7943d29cce97d1a32c591d (DEFAULT_TABLE table=test-table [id=0e02081877674d12abf0b25781092593]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:29.437924 28526 tablet_service.cc:1467] Processing CreateTablet for tablet 141355250c7943d29cce97d1a32c591d (DEFAULT_TABLE table=test-table [id=0e02081877674d12abf0b25781092593]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:29.438108 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.432404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34078 (local address 127.8.88.131:32961)
0928 13:12:29.432552 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:29.432557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:29.432604 (+ 47us) server_negotiation.cc:407] Connection header received
0928 13:12:29.432852 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:29.432856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:29.432897 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:29.432954 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:29.434383 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.435311 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.436394 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.437627 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.437693 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.437775 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:29.437987 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 13:12:29.438031 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260928 13:12:29.438309 28526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141355250c7943d29cce97d1a32c591d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:29.438310 28395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141355250c7943d29cce97d1a32c591d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:29.440937 28738 tablet_bootstrap.cc:492] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc: Bootstrap starting.
I20260928 13:12:29.440982 28657 tablet_service.cc:1467] Processing CreateTablet for tablet 141355250c7943d29cce97d1a32c591d (DEFAULT_TABLE table=test-table [id=0e02081877674d12abf0b25781092593]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 13:12:29.441306 28657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 141355250c7943d29cce97d1a32c591d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:29.441668 28738 tablet_bootstrap.cc:654] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:29.441955 28738 log.cc:824] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:29.442270 28737 tablet_bootstrap.cc:492] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb: Bootstrap starting.
I20260928 13:12:29.442806 28738 tablet_bootstrap.cc:492] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc: No bootstrap required, opened a new log
I20260928 13:12:29.442917 28738 ts_tablet_manager.cc:1397] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 13:12:29.442932 28737 tablet_bootstrap.cc:654] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:29.443245 28737 log.cc:824] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:29.444093 28737 tablet_bootstrap.cc:492] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb: No bootstrap required, opened a new log
I20260928 13:12:29.444162 28737 ts_tablet_manager.cc:1397] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:12:29.444423 28741 tablet_bootstrap.cc:492] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56: Bootstrap starting.
I20260928 13:12:29.445070 28741 tablet_bootstrap.cc:654] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:29.445385 28741 log.cc:824] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:29.445753 28738 raft_consensus.cc:348] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9702ad6b01254102af5a1ae47f742b56" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 32961 } } peers { permanent_uuid: "a44216177276449ea0220332cb9ce0bc" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39791 } } peers { permanent_uuid: "0c4ae81755704c9c92b6501b65bbc9bb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 33577 } }
I20260928 13:12:29.445928 28738 raft_consensus.cc:374] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:29.445963 28738 raft_consensus.cc:729] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a44216177276449ea0220332cb9ce0bc, State: Initialized, Role: FOLLOWER
I20260928 13:12:29.446115 28738 consensus_queue.cc:260] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9702ad6b01254102af5a1ae47f742b56" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 32961 } } peers { permanent_uuid: "a44216177276449ea0220332cb9ce0bc" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39791 } } peers { permanent_uuid: "0c4ae81755704c9c92b6501b65bbc9bb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 33577 } }
I20260928 13:12:29.446282 28741 tablet_bootstrap.cc:492] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56: No bootstrap required, opened a new log
I20260928 13:12:29.446357 28741 ts_tablet_manager.cc:1397] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 13:12:29.446386 28738 ts_tablet_manager.cc:1428] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260928 13:12:29.446519 28592 heartbeater.cc:503] Master 127.8.88.190:39061 was elected leader, sending a full tablet report...
I20260928 13:12:29.446914 28737 raft_consensus.cc:348] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9702ad6b01254102af5a1ae47f742b56" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 32961 } } peers { permanent_uuid: "a44216177276449ea0220332cb9ce0bc" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39791 } } peers { permanent_uuid: "0c4ae81755704c9c92b6501b65bbc9bb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 33577 } }
I20260928 13:12:29.447089 28737 raft_consensus.cc:374] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:29.447124 28737 raft_consensus.cc:729] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c4ae81755704c9c92b6501b65bbc9bb, State: Initialized, Role: FOLLOWER
I20260928 13:12:29.447228 28737 consensus_queue.cc:260] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9702ad6b01254102af5a1ae47f742b56" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 32961 } } peers { permanent_uuid: "a44216177276449ea0220332cb9ce0bc" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39791 } } peers { permanent_uuid: "0c4ae81755704c9c92b6501b65bbc9bb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 33577 } }
I20260928 13:12:29.447494 28737 ts_tablet_manager.cc:1428] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 13:12:29.447513 28461 heartbeater.cc:503] Master 127.8.88.190:39061 was elected leader, sending a full tablet report...
I20260928 13:12:29.449213 28741 raft_consensus.cc:348] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9702ad6b01254102af5a1ae47f742b56" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 32961 } } peers { permanent_uuid: "a44216177276449ea0220332cb9ce0bc" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39791 } } peers { permanent_uuid: "0c4ae81755704c9c92b6501b65bbc9bb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 33577 } }
I20260928 13:12:29.449348 28741 raft_consensus.cc:374] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:29.449370 28741 raft_consensus.cc:729] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9702ad6b01254102af5a1ae47f742b56, State: Initialized, Role: FOLLOWER
I20260928 13:12:29.449486 28741 consensus_queue.cc:260] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9702ad6b01254102af5a1ae47f742b56" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 32961 } } peers { permanent_uuid: "a44216177276449ea0220332cb9ce0bc" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39791 } } peers { permanent_uuid: "0c4ae81755704c9c92b6501b65bbc9bb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 33577 } }
I20260928 13:12:29.449753 28741 ts_tablet_manager.cc:1428] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 13:12:29.449818 28723 heartbeater.cc:503] Master 127.8.88.190:39061 was elected leader, sending a full tablet report...
I20260928 13:12:29.460259 28745 raft_consensus.cc:482] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 13:12:29.460412 28745 raft_consensus.cc:504] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9702ad6b01254102af5a1ae47f742b56" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 32961 } } peers { permanent_uuid: "a44216177276449ea0220332cb9ce0bc" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39791 } } peers { permanent_uuid: "0c4ae81755704c9c92b6501b65bbc9bb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 33577 } }
I20260928 13:12:29.460803 28745 leader_election.cc:288] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a44216177276449ea0220332cb9ce0bc (127.8.88.130:39791), 0c4ae81755704c9c92b6501b65bbc9bb (127.8.88.129:33577)
I20260928 13:12:29.463919 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.460982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.130:39791 (local address 127.8.88.131:40909)
0928 13:12:29.461038 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:29.461049 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:29.461105 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:29.461252 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:29.461253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:29.461255 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:29.461334 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.461336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.462240 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.462242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.463076 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.463078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.463663 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.463665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.463743 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.463747 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:12:29.463803 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 13:12:29.463854 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 13:12:29.463979 28735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.461043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:40909 (local address 127.8.88.130:39791)
0928 13:12:29.461096 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:29.461098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:29.461107 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:12:29.461171 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:29.461172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:29.461181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:29.461226 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:29.461411 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.462105 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.463171 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.463576 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.463632 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.463639 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:29.463848 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 13:12:29.463893 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 13:12:29.464078 28746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.460992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:33577 (local address 127.8.88.131:52883)
0928 13:12:29.461180 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:29.461190 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:29.461285 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:29.461432 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:29.461432 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:29.461434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 13:12:29.461508 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.461509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.462370 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.462372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.463157 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:29.463159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.463810 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.463811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:29.463891 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.463895 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 13:12:29.463955 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 13:12:29.463991 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":111,"threads_started":1}
I20260928 13:12:29.464104 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.461065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.88.131:52883 (local address 127.8.88.129:33577)
0928 13:12:29.461112 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:29.461115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:29.461263 (+ 148us) server_negotiation.cc:407] Connection header received
0928 13:12:29.461350 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:29.461351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:29.461367 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:29.461429 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 13:12:29.461588 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.462262 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.463265 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.463699 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.463783 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.463793 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:29.463974 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 13:12:29.464024 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:12:29.464569 28546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141355250c7943d29cce97d1a32c591d" candidate_uuid: "9702ad6b01254102af5a1ae47f742b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44216177276449ea0220332cb9ce0bc" is_pre_election: true
I20260928 13:12:29.464566 28415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141355250c7943d29cce97d1a32c591d" candidate_uuid: "9702ad6b01254102af5a1ae47f742b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4ae81755704c9c92b6501b65bbc9bb" is_pre_election: true
I20260928 13:12:29.464736 28546 raft_consensus.cc:2454] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9702ad6b01254102af5a1ae47f742b56 in term 0.
I20260928 13:12:29.464771 28415 raft_consensus.cc:2454] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9702ad6b01254102af5a1ae47f742b56 in term 0.
I20260928 13:12:29.465000 28613 leader_election.cc:302] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9702ad6b01254102af5a1ae47f742b56, a44216177276449ea0220332cb9ce0bc; no voters:
I20260928 13:12:29.465144 28745 raft_consensus.cc:2781] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 13:12:29.465199 28745 raft_consensus.cc:482] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 13:12:29.465257 28745 raft_consensus.cc:3037] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:29.465982 28745 raft_consensus.cc:504] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9702ad6b01254102af5a1ae47f742b56" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 32961 } } peers { permanent_uuid: "a44216177276449ea0220332cb9ce0bc" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39791 } } peers { permanent_uuid: "0c4ae81755704c9c92b6501b65bbc9bb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 33577 } }
I20260928 13:12:29.466184 28745 leader_election.cc:288] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [CANDIDATE]: Term 1 election: Requested vote from peers a44216177276449ea0220332cb9ce0bc (127.8.88.130:39791), 0c4ae81755704c9c92b6501b65bbc9bb (127.8.88.129:33577)
I20260928 13:12:29.466260 28546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141355250c7943d29cce97d1a32c591d" candidate_uuid: "9702ad6b01254102af5a1ae47f742b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a44216177276449ea0220332cb9ce0bc"
I20260928 13:12:29.466316 28415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "141355250c7943d29cce97d1a32c591d" candidate_uuid: "9702ad6b01254102af5a1ae47f742b56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c4ae81755704c9c92b6501b65bbc9bb"
I20260928 13:12:29.466344 28546 raft_consensus.cc:3037] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:29.466406 28415 raft_consensus.cc:3037] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:29.467089 28546 raft_consensus.cc:2454] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9702ad6b01254102af5a1ae47f742b56 in term 1.
I20260928 13:12:29.467317 28613 leader_election.cc:302] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9702ad6b01254102af5a1ae47f742b56, a44216177276449ea0220332cb9ce0bc; no voters:
I20260928 13:12:29.467470 28745 raft_consensus.cc:2781] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:29.467659 28745 raft_consensus.cc:686] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [term 1 LEADER]: Becoming Leader. State: Replica: 9702ad6b01254102af5a1ae47f742b56, State: Running, Role: LEADER
I20260928 13:12:29.467597 28415 raft_consensus.cc:2454] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9702ad6b01254102af5a1ae47f742b56 in term 1.
I20260928 13:12:29.467767 28745 consensus_queue.cc:237] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9702ad6b01254102af5a1ae47f742b56" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 32961 } } peers { permanent_uuid: "a44216177276449ea0220332cb9ce0bc" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39791 } } peers { permanent_uuid: "0c4ae81755704c9c92b6501b65bbc9bb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 33577 } }
I20260928 13:12:29.468699 28273 catalog_manager.cc:5495] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9702ad6b01254102af5a1ae47f742b56 (127.8.88.131). New cstate: current_term: 1 leader_uuid: "9702ad6b01254102af5a1ae47f742b56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9702ad6b01254102af5a1ae47f742b56" member_type: VOTER last_known_addr { host: "127.8.88.131" port: 32961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a44216177276449ea0220332cb9ce0bc" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c4ae81755704c9c92b6501b65bbc9bb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 33577 } health_report { overall_health: UNKNOWN } } }
I20260928 13:12:29.482954 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:29.480173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34090 (local address 127.8.88.131:32961)
0928 13:12:29.480246 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:29.480248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:29.480258 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:12:29.480310 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:29.480310 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:29.480327 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:29.480396 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:29.480536 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.481131 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.482022 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:29.482251 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:29.482323 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:29.482520 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:29.482629 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:29.482783 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:29.482803 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 13:12:29.482822 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 13:12:29.485126 28546 raft_consensus.cc:1261] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc [term 1 FOLLOWER]: Refusing update from remote peer 9702ad6b01254102af5a1ae47f742b56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:29.485379 28415 raft_consensus.cc:1261] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb [term 1 FOLLOWER]: Refusing update from remote peer 9702ad6b01254102af5a1ae47f742b56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:29.485586 28745 consensus_queue.cc:1035] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a44216177276449ea0220332cb9ce0bc" member_type: VOTER last_known_addr { host: "127.8.88.130" port: 39791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:29.485736 28747 consensus_queue.cc:1035] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c4ae81755704c9c92b6501b65bbc9bb" member_type: VOTER last_known_addr { host: "127.8.88.129" port: 33577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:29.488408 28750 mvcc.cc:205] Tried to move back new op lower bound from 7334302308287684608 to 7334302308220735488. Current Snapshot: MvccSnapshot[applied={T|T < 7334302308287684608}]
I20260928 13:12:29.489094 28752 mvcc.cc:205] Tried to move back new op lower bound from 7334302308287684608 to 7334302308220735488. Current Snapshot: MvccSnapshot[applied={T|T < 7334302308287684608}]
I20260928 13:12:29.489624 28751 mvcc.cc:205] Tried to move back new op lower bound from 7334302308287684608 to 7334302308220735488. Current Snapshot: MvccSnapshot[applied={T|T < 7334302308287684608 or (T in {7334302308287684608})}]
I20260928 13:12:29.491717 28272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:12:29.491829 28272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 13:12:29.492604 28272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ace4024e8964789a19156ce8110a3bb: Sending DeleteTablet for 3 replicas of tablet 141355250c7943d29cce97d1a32c591d
I20260928 13:12:29.493021 28657 tablet_service.cc:1514] Processing DeleteTablet for tablet 141355250c7943d29cce97d1a32c591d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:29 UTC) from {username='slave'} at 127.0.0.1:34078
I20260928 13:12:29.493088 28395 tablet_service.cc:1514] Processing DeleteTablet for tablet 141355250c7943d29cce97d1a32c591d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:29 UTC) from {username='slave'} at 127.0.0.1:49480
I20260928 13:12:29.493382 28526 tablet_service.cc:1514] Processing DeleteTablet for tablet 141355250c7943d29cce97d1a32c591d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 13:12:29 UTC) from {username='slave'} at 127.0.0.1:51222
I20260928 13:12:29.493440 28760 tablet_replica.cc:330] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56: stopping tablet replica
I20260928 13:12:29.493534 28760 raft_consensus.cc:2229] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [term 1 LEADER]: Raft consensus shutting down.
I20260928 13:12:29.493613 28762 tablet_replica.cc:330] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc: stopping tablet replica
I20260928 13:12:29.493340 28761 tablet_replica.cc:330] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb: stopping tablet replica
I20260928 13:12:29.493664 28760 raft_consensus.cc:2258] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:12:29.493710 28762 raft_consensus.cc:2229] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:12:29.493782 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 28333
I20260928 13:12:29.493873 28761 raft_consensus.cc:2229] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 13:12:29.493989 28761 raft_consensus.cc:2258] T 141355250c7943d29cce97d1a32c591d P 0c4ae81755704c9c92b6501b65bbc9bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:12:29.493822 28762 raft_consensus.cc:2258] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 13:12:29.494789 28762 ts_tablet_manager.cc:1905] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:12:29.494994 28760 ts_tablet_manager.cc:1905] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 13:12:29.496562 28762 ts_tablet_manager.cc:1918] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:12:29.496667 28762 log.cc:1196] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-1/wal/wals/141355250c7943d29cce97d1a32c591d
I20260928 13:12:29.496989 28762 ts_tablet_manager.cc:1939] T 141355250c7943d29cce97d1a32c591d P a44216177276449ea0220332cb9ce0bc: Deleting consensus metadata
I20260928 13:12:29.497006 28760 ts_tablet_manager.cc:1918] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 13:12:29.497072 28760 log.cc:1196] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56: Deleting WAL directory at /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790600874459249-8546-0/minicluster-data/ts-2/wal/wals/141355250c7943d29cce97d1a32c591d
I20260928 13:12:29.497319 28760 ts_tablet_manager.cc:1939] T 141355250c7943d29cce97d1a32c591d P 9702ad6b01254102af5a1ae47f742b56: Deleting consensus metadata
I20260928 13:12:29.497534 28260 catalog_manager.cc:4841] TS a44216177276449ea0220332cb9ce0bc (127.8.88.130:39791): tablet 141355250c7943d29cce97d1a32c591d (table test-table [id=0e02081877674d12abf0b25781092593]) successfully deleted
I20260928 13:12:29.497964 28259 catalog_manager.cc:4841] TS 9702ad6b01254102af5a1ae47f742b56 (127.8.88.131:32961): tablet 141355250c7943d29cce97d1a32c591d (table test-table [id=0e02081877674d12abf0b25781092593]) successfully deleted
W20260928 13:12:29.499625 28259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.88.129:33577 (error 108)
W20260928 13:12:29.500900 28731 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:29.500030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.129:33577 (local address 127.0.0.1:49482)
0928 13:12:29.500099 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:29.500112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:29.500157 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:29.500809 (+ 652us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.129:33577: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 13:12:29.501132 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 28464
W20260928 13:12:29.501140 28259 catalog_manager.cc:4584] TS 0c4ae81755704c9c92b6501b65bbc9bb (127.8.88.129:33577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 141355250c7943d29cce97d1a32c591d: Network error: Client connection negotiation failed: client connection to 127.8.88.129:33577: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 13:12:29.506629 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 28595
I20260928 13:12:29.512039 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 28242
2026-09-28T13:12:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (685 ms)
[----------] 6 tests from EncryptionPolicyTest (8444 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:29 dist-test-slave-p1lc krb5kdc[28766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:29 dist-test-slave-p1lc krb5kdc[28766](info): set up 2 sockets
Sep 28 13:12:29 dist-test-slave-p1lc krb5kdc[28766](info): commencing operation
krb5kdc: starting...
W20260928 13:12:30.577145 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:30 dist-test-slave-p1lc krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:12:30.708360 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:30.792240 28779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:30.792529 28779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:30.792575 28779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:30.793931 28779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:30.794018 28779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:30.794044 28779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:30.794065 28779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:30.794094 28779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:30.795784 28779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:30.796100 28779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:30.796355 28779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:30.802138 28784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:30.802138 28785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:30.802426 28779 server_base.cc:1034] running on GCE node
W20260928 13:12:30.802138 28787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:30.802812 28779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:30.803143 28779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:30.804330 28779 hybrid_clock.cc:648] HybridClock initialized: now 1790601150804310 us; error 42 us; skew 500 ppm
Sep 28 13:12:30 dist-test-slave-p1lc krb5kdc[28766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:30.807251 28779 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:12:30.807787 28779 webserver.cc:460] Webserver started at http://127.0.0.1:46109/ using document root <none> and password file <none>
I20260928 13:12:30.808037 28779 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:30.808086 28779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:30.808244 28779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:30.809448 28779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "ddcdc003fe014fbc9dceb1232cc77adf"
format_stamp: "Formatted at 2026-09-28 13:12:30 on dist-test-slave-p1lc"
server_key: "9fb557958fa84368d36e90dfe2227520"
server_key_iv: "ddb65ba3ee7a2d997be0958a929acd37"
server_key_version: "encryptionkey@0"
I20260928 13:12:30.809852 28779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "ddcdc003fe014fbc9dceb1232cc77adf"
format_stamp: "Formatted at 2026-09-28 13:12:30 on dist-test-slave-p1lc"
server_key: "9fb557958fa84368d36e90dfe2227520"
server_key_iv: "ddb65ba3ee7a2d997be0958a929acd37"
server_key_version: "encryptionkey@0"
I20260928 13:12:30.811560 28779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 13:12:30.812521 28794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:30.812740 28779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:30.812934 28779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "ddcdc003fe014fbc9dceb1232cc77adf"
format_stamp: "Formatted at 2026-09-28 13:12:30 on dist-test-slave-p1lc"
server_key: "9fb557958fa84368d36e90dfe2227520"
server_key_iv: "ddb65ba3ee7a2d997be0958a929acd37"
server_key_version: "encryptionkey@0"
I20260928 13:12:30.813053 28779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:30.832515 28779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:30.835081 28779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:30.835260 28779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:30.840171 28779 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41663
I20260928 13:12:30.840227 28846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41663 every 8 connection(s)
I20260928 13:12:30.840600 28779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:30.841737 28847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:30.844108 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 28779
I20260928 13:12:30.844213 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:30.844271 28847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf: Bootstrap starting.
I20260928 13:12:30.844455 8546 external_mini_cluster.cc:1428] Setting key b59f7dbfa5826942f944baf5c8085f0a
I20260928 13:12:30.845008 28847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:30.845376 28847 log.cc:824] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:30.846166 28847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf: No bootstrap required, opened a new log
I20260928 13:12:30.848474 28847 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddcdc003fe014fbc9dceb1232cc77adf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } }
I20260928 13:12:30.848665 28847 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:30.848699 28847 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddcdc003fe014fbc9dceb1232cc77adf, State: Initialized, Role: FOLLOWER
I20260928 13:12:30.848830 28847 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddcdc003fe014fbc9dceb1232cc77adf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } }
I20260928 13:12:30.848930 28847 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:30.848968 28847 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:30.849015 28847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:30.849665 28847 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddcdc003fe014fbc9dceb1232cc77adf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } }
I20260928 13:12:30.849787 28847 leader_election.cc:302] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddcdc003fe014fbc9dceb1232cc77adf; no voters:
I20260928 13:12:30.849987 28847 leader_election.cc:288] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:30.850061 28852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:30.850201 28852 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [term 1 LEADER]: Becoming Leader. State: Replica: ddcdc003fe014fbc9dceb1232cc77adf, State: Running, Role: LEADER
I20260928 13:12:30.850332 28847 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:30.850306 28852 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddcdc003fe014fbc9dceb1232cc77adf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } }
I20260928 13:12:30.850757 28853 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddcdc003fe014fbc9dceb1232cc77adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddcdc003fe014fbc9dceb1232cc77adf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } } }
I20260928 13:12:30.850747 28852 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddcdc003fe014fbc9dceb1232cc77adf. Latest consensus state: current_term: 1 leader_uuid: "ddcdc003fe014fbc9dceb1232cc77adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddcdc003fe014fbc9dceb1232cc77adf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41663 } } }
I20260928 13:12:30.850891 28853 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:30.850907 28852 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:30.851863 28858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:30.852415 28858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 13:12:30 dist-test-slave-p1lc krb5kdc[28766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 13:12:30.854195 28858 catalog_manager.cc:1347] Generated new cluster ID: b9e7fed6f36c46a4b4b1f9b9dca4155f
I20260928 13:12:30.854249 28858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:30.857659 28850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:30.845082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41177 (local address 127.0.0.1:41663)
0928 13:12:30.845263 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:30.845271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:30.845304 (+ 33us) server_negotiation.cc:407] Connection header received
0928 13:12:30.845945 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:30.845953 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:30.846005 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:30.846126 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:30.846980 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:30.848024 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:30.849168 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:30.849608 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:30.849691 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:30.853437 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:30.853442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:30.853442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:30.853455 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:30.855732 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:30.856338 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:30.856341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:30.856342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:30.856450 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:30.856699 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:30.856700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:30.856700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:30.857026 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:30.857179 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:30.857352 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 13:12:30.857452 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260928 13:12:30.858717 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:30.865051 28858 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:30.866230 28858 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:30.873157 28850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:30.864546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57399 (local address 127.0.0.1:41663)
0928 13:12:30.864589 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:30.864591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:30.864602 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:12:30.864629 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:30.864629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:30.864640 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:30.864712 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:30.865850 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:30.866809 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:30.867993 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:30.868343 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:30.868420 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:30.870093 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:30.870095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:30.870095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:30.870103 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:30.871765 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:30.872229 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:30.872230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:30.872230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:30.872290 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:30.872533 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:30.872534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:30.872534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:30.872663 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:30.872766 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:30.872984 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 13:12:30.873054 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":22}
I20260928 13:12:30.881820 28858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddcdc003fe014fbc9dceb1232cc77adf: Generated new TSK 0
I20260928 13:12:30.882056 28858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:30.889532 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 28779
W20260928 13:12:30.894551 28764 connection.cc:535] client connection to 127.0.0.1:41663 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41663): Connection reset by peer (error 104)
2026-09-28T13:12:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:30 dist-test-slave-p1lc krb5kdc[28878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:30 dist-test-slave-p1lc krb5kdc[28878](info): set up 2 sockets
Sep 28 13:12:30 dist-test-slave-p1lc krb5kdc[28878](info): commencing operation
krb5kdc: starting...
W20260928 13:12:31.945207 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:32 dist-test-slave-p1lc krb5kdc[28878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:12:32.068516 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:32.150792 28891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:32.151060 28891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:32.151100 28891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:32.152315 28891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:32.152395 28891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:32.152421 28891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:32.152442 28891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:32.152472 28891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:32.153993 28891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38599
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:32.154296 28891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:32.154537 28891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:32.159773 28896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:32.159781 28897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:32.159860 28891 server_base.cc:1034] running on GCE node
W20260928 13:12:32.159839 28899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:32.160249 28891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:32.160506 28891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:32.161655 28891 hybrid_clock.cc:648] HybridClock initialized: now 1790601152161633 us; error 39 us; skew 500 ppm
Sep 28 13:12:32 dist-test-slave-p1lc krb5kdc[28878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:32.164175 28891 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:12:32.164649 28891 webserver.cc:460] Webserver started at http://127.0.0.1:33165/ using document root <none> and password file <none>
I20260928 13:12:32.164942 28891 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:32.164994 28891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:32.165086 28891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:32.166126 28891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "59eb5ecce0c0456f8a6a326b27ee9331"
format_stamp: "Formatted at 2026-09-28 13:12:32 on dist-test-slave-p1lc"
server_key: "0e1b89af5aaf7b5bfbae38daf2720811"
server_key_iv: "9c6ed09893b43c879df779dad23ccb73"
server_key_version: "encryptionkey@0"
I20260928 13:12:32.166441 28891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "59eb5ecce0c0456f8a6a326b27ee9331"
format_stamp: "Formatted at 2026-09-28 13:12:32 on dist-test-slave-p1lc"
server_key: "0e1b89af5aaf7b5bfbae38daf2720811"
server_key_iv: "9c6ed09893b43c879df779dad23ccb73"
server_key_version: "encryptionkey@0"
I20260928 13:12:32.167760 28891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:12:32.168524 28906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:32.168826 28891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:12:32.168939 28891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "59eb5ecce0c0456f8a6a326b27ee9331"
format_stamp: "Formatted at 2026-09-28 13:12:32 on dist-test-slave-p1lc"
server_key: "0e1b89af5aaf7b5bfbae38daf2720811"
server_key_iv: "9c6ed09893b43c879df779dad23ccb73"
server_key_version: "encryptionkey@0"
I20260928 13:12:32.169016 28891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:32.212939 28891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:32.215103 28891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:32.215219 28891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:32.219324 28891 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38599
I20260928 13:12:32.219414 28958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38599 every 8 connection(s)
I20260928 13:12:32.219772 28891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:32.220852 28959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:32.223251 28959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331: Bootstrap starting.
I20260928 13:12:32.223783 28959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:32.224040 28959 log.cc:824] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:32.224218 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 28891
I20260928 13:12:32.224363 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:32.224560 8546 external_mini_cluster.cc:1428] Setting key 2431a38570855171d18412f0d858223b
I20260928 13:12:32.224931 28959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331: No bootstrap required, opened a new log
I20260928 13:12:32.226691 28959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eb5ecce0c0456f8a6a326b27ee9331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38599 } }
I20260928 13:12:32.226843 28959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:32.226866 28959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59eb5ecce0c0456f8a6a326b27ee9331, State: Initialized, Role: FOLLOWER
I20260928 13:12:32.227025 28959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eb5ecce0c0456f8a6a326b27ee9331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38599 } }
I20260928 13:12:32.227098 28959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:32.227142 28959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:32.227236 28959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:32.227797 28959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eb5ecce0c0456f8a6a326b27ee9331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38599 } }
I20260928 13:12:32.227921 28959 leader_election.cc:302] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59eb5ecce0c0456f8a6a326b27ee9331; no voters:
I20260928 13:12:32.228132 28959 leader_election.cc:288] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:32.228153 28964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:32.228262 28964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [term 1 LEADER]: Becoming Leader. State: Replica: 59eb5ecce0c0456f8a6a326b27ee9331, State: Running, Role: LEADER
I20260928 13:12:32.228351 28964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eb5ecce0c0456f8a6a326b27ee9331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38599 } }
I20260928 13:12:32.228384 28959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:32.230371 28965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59eb5ecce0c0456f8a6a326b27ee9331. Latest consensus state: current_term: 1 leader_uuid: "59eb5ecce0c0456f8a6a326b27ee9331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eb5ecce0c0456f8a6a326b27ee9331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38599 } } }
I20260928 13:12:32.230211 28966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59eb5ecce0c0456f8a6a326b27ee9331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eb5ecce0c0456f8a6a326b27ee9331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38599 } } }
I20260928 13:12:32.230455 28966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:32.230455 28965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331 [sys.catalog]: This master's current role is: LEADER
W20260928 13:12:32.231024 28978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:12:32.231114 28978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:12:32.231172 28981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:32.231340 28981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:32.232229 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:32.225097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44405 (local address 127.0.0.1:38599)
0928 13:12:32.225283 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:32.225293 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:32.225331 (+ 38us) server_negotiation.cc:407] Connection header received
0928 13:12:32.226698 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:32.226756 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:32.226813 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:32.226993 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:32.227196 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:32.228769 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:32.230005 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:32.230694 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:32.230818 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:32.231581 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:32.231584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:32.231585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:32.231762 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:32.231792 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:32.231803 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:32.231896 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:32.231985 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 13:12:32.232014 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260928 13:12:32.233120 28981 catalog_manager.cc:1347] Generated new cluster ID: 26423744d5864120ab02aa2737d6ba3f
I20260928 13:12:32.233174 28981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:32.233167 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:32.238341 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:32.234763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50251 (local address 127.0.0.1:38599)
0928 13:12:32.234794 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:32.234795 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:32.234812 (+ 17us) server_negotiation.cc:407] Connection header received
0928 13:12:32.234869 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:32.234870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:32.234878 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:32.234921 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:32.235087 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:32.235811 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:32.236893 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:32.237228 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:32.237285 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:32.238071 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:32.238072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:32.238072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:32.238077 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:32.238090 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:32.238095 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:32.238150 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:32.238257 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 13:12:32.238269 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 13:12:32.247843 28981 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:32.248402 28981 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:32.255183 28981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59eb5ecce0c0456f8a6a326b27ee9331: Generated new TSK 0
I20260928 13:12:32.255435 28981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:32.260790 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 28891
W20260928 13:12:32.265082 28876 connection.cc:535] client connection to 127.0.0.1:38599 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38599): Connection reset by peer (error 104)
2026-09-28T13:12:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:32 dist-test-slave-p1lc krb5kdc[28991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:32 dist-test-slave-p1lc krb5kdc[28991](info): set up 2 sockets
Sep 28 13:12:32 dist-test-slave-p1lc krb5kdc[28991](info): commencing operation
krb5kdc: starting...
W20260928 13:12:33.305241 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:33 dist-test-slave-p1lc krb5kdc[28991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:12:33.431249 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:33.513676 29004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:33.513902 29004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:33.513927 29004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:33.515163 29004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:33.515213 29004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:33.515228 29004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:33.515240 29004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:33.515252 29004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:33.516821 29004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:33.517104 29004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:33.517331 29004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:33.522408 29010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:33.522416 29009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:33.522416 29012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:33.522454 29004 server_base.cc:1034] running on GCE node
I20260928 13:12:33.522840 29004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:33.523159 29004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:33.524304 29004 hybrid_clock.cc:648] HybridClock initialized: now 1790601153524290 us; error 32 us; skew 500 ppm
Sep 28 13:12:33 dist-test-slave-p1lc krb5kdc[28991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:33.526988 29004 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:12:33.527452 29004 webserver.cc:460] Webserver started at http://127.0.0.1:44875/ using document root <none> and password file <none>
I20260928 13:12:33.527683 29004 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:33.527748 29004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:33.527902 29004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:33.529036 29004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "64ed9f02115e4487a49fbed3d9207325"
format_stamp: "Formatted at 2026-09-28 13:12:33 on dist-test-slave-p1lc"
server_key: "1a091b8727e814354308cb974bda16d5"
server_key_iv: "299733fb0b2ef384e518e12f84ae18ee"
server_key_version: "encryptionkey@0"
I20260928 13:12:33.529392 29004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "64ed9f02115e4487a49fbed3d9207325"
format_stamp: "Formatted at 2026-09-28 13:12:33 on dist-test-slave-p1lc"
server_key: "1a091b8727e814354308cb974bda16d5"
server_key_iv: "299733fb0b2ef384e518e12f84ae18ee"
server_key_version: "encryptionkey@0"
I20260928 13:12:33.530714 29004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:12:33.531531 29019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:33.531812 29004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:33.531929 29004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "64ed9f02115e4487a49fbed3d9207325"
format_stamp: "Formatted at 2026-09-28 13:12:33 on dist-test-slave-p1lc"
server_key: "1a091b8727e814354308cb974bda16d5"
server_key_iv: "299733fb0b2ef384e518e12f84ae18ee"
server_key_version: "encryptionkey@0"
I20260928 13:12:33.532006 29004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:33.535430 29004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:33.537694 29004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:33.537846 29004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:33.542316 29004 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46769
I20260928 13:12:33.542400 29071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46769 every 8 connection(s)
I20260928 13:12:33.542739 29004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:33.543900 29072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:33.546420 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 29004
I20260928 13:12:33.546500 29072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325: Bootstrap starting.
I20260928 13:12:33.546567 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:33.546759 8546 external_mini_cluster.cc:1428] Setting key 302331ad0dc23e1f6922e1bd61f03cff
I20260928 13:12:33.547224 29072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:33.547621 29072 log.cc:824] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:33.548431 29072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325: No bootstrap required, opened a new log
I20260928 13:12:33.550446 29072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ed9f02115e4487a49fbed3d9207325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46769 } }
I20260928 13:12:33.550607 29072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:33.550631 29072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ed9f02115e4487a49fbed3d9207325, State: Initialized, Role: FOLLOWER
I20260928 13:12:33.550787 29072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ed9f02115e4487a49fbed3d9207325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46769 } }
I20260928 13:12:33.550890 29072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:33.550916 29072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:33.550946 29072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:33.551505 29072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ed9f02115e4487a49fbed3d9207325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46769 } }
I20260928 13:12:33.551594 29072 leader_election.cc:302] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64ed9f02115e4487a49fbed3d9207325; no voters:
I20260928 13:12:33.551728 29072 leader_election.cc:288] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:33.551826 29077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:33.552018 29072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:33.552002 29077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [term 1 LEADER]: Becoming Leader. State: Replica: 64ed9f02115e4487a49fbed3d9207325, State: Running, Role: LEADER
I20260928 13:12:33.552253 29077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ed9f02115e4487a49fbed3d9207325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46769 } }
I20260928 13:12:33.552654 29078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64ed9f02115e4487a49fbed3d9207325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ed9f02115e4487a49fbed3d9207325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46769 } } }
I20260928 13:12:33.552763 29078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:33.553189 29079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64ed9f02115e4487a49fbed3d9207325. Latest consensus state: current_term: 1 leader_uuid: "64ed9f02115e4487a49fbed3d9207325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64ed9f02115e4487a49fbed3d9207325" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46769 } } }
I20260928 13:12:33.553284 29079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325 [sys.catalog]: This master's current role is: LEADER
W20260928 13:12:33.554046 29093 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:12:33.554137 29093 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:12:33.554242 29086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:33.554265 29075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:33.547311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35617 (local address 127.0.0.1:46769)
0928 13:12:33.547513 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:33.547521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:33.547547 (+ 26us) server_negotiation.cc:407] Connection header received
0928 13:12:33.548124 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:33.548132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:33.548159 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:33.548297 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:33.553654 (+ 5357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:33.553658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:33.553659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:33.553665 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:33.553714 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:33.553731 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:33.553898 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:33.553976 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 13:12:33.554021 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
I20260928 13:12:33.554431 29086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:33.555251 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:33.556058 29086 catalog_manager.cc:1347] Generated new cluster ID: 866ca601feba4f368101e3fe8092c490
I20260928 13:12:33.556111 29086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:33.562273 29075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:33.556970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39939 (local address 127.0.0.1:46769)
0928 13:12:33.557007 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:33.557009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:33.561143 (+ 4134us) server_negotiation.cc:407] Connection header received
0928 13:12:33.561202 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:33.561203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:33.561210 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:33.561285 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:33.562029 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:33.562031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:33.562031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:33.562036 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:33.562055 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:33.562067 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:33.562132 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:33.562182 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 13:12:33.562196 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 13:12:33.564774 29086 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:33.564968 29086 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:33.571106 29086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64ed9f02115e4487a49fbed3d9207325: Generated new TSK 0
I20260928 13:12:33.571298 29086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:33.574349 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 29004
2026-09-28T13:12:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:33 dist-test-slave-p1lc krb5kdc[29103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:33 dist-test-slave-p1lc krb5kdc[29103](info): set up 2 sockets
Sep 28 13:12:33 dist-test-slave-p1lc krb5kdc[29103](info): commencing operation
krb5kdc: starting...
W20260928 13:12:34.616806 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:34 dist-test-slave-p1lc krb5kdc[29103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:12:34.743541 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:34.826640 29116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:34.826860 29116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:34.826884 29116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:34.828068 29116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:34.828111 29116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:34.828125 29116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:34.828138 29116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:34.828150 29116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:34.830233 29116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:34.830447 29116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:34.830624 29116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:12:34.835908 29116 server_base.cc:1034] running on GCE node
W20260928 13:12:34.835856 29122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:34.835911 29121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:34.835888 29124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:34.836402 29116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:34.836668 29116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:34.837903 29116 hybrid_clock.cc:648] HybridClock initialized: now 1790601154837875 us; error 57 us; skew 500 ppm
Sep 28 13:12:34 dist-test-slave-p1lc krb5kdc[29103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:34.840718 29116 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:12:34.841355 29116 webserver.cc:460] Webserver started at http://127.0.0.1:42539/ using document root <none> and password file <none>
I20260928 13:12:34.841624 29116 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:34.841703 29116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:34.841886 29116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:34.843039 29116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "f2c8fe7eebdc4127b4d583dab44faba6"
format_stamp: "Formatted at 2026-09-28 13:12:34 on dist-test-slave-p1lc"
server_key: "150a982579b758246c18b3890202c1ca"
server_key_iv: "0272993556d1d7d0052805fec5f2cf62"
server_key_version: "encryptionkey@0"
I20260928 13:12:34.843416 29116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "f2c8fe7eebdc4127b4d583dab44faba6"
format_stamp: "Formatted at 2026-09-28 13:12:34 on dist-test-slave-p1lc"
server_key: "150a982579b758246c18b3890202c1ca"
server_key_iv: "0272993556d1d7d0052805fec5f2cf62"
server_key_version: "encryptionkey@0"
I20260928 13:12:34.844910 29116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:34.845674 29131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:34.845845 29116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:12:34.845945 29116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "f2c8fe7eebdc4127b4d583dab44faba6"
format_stamp: "Formatted at 2026-09-28 13:12:34 on dist-test-slave-p1lc"
server_key: "150a982579b758246c18b3890202c1ca"
server_key_iv: "0272993556d1d7d0052805fec5f2cf62"
server_key_version: "encryptionkey@0"
I20260928 13:12:34.846032 29116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:34.864988 29116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:34.868096 29116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:34.868221 29116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:34.872553 29116 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45243
I20260928 13:12:34.872620 29183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45243 every 8 connection(s)
I20260928 13:12:34.873045 29116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:34.874147 29184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:34.876734 29184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6: Bootstrap starting.
I20260928 13:12:34.877493 29184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:34.877822 29184 log.cc:824] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:34.878628 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 29116
I20260928 13:12:34.878743 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:34.878886 29184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6: No bootstrap required, opened a new log
I20260928 13:12:34.878962 8546 external_mini_cluster.cc:1428] Setting key 3f20b20f539d720e463299a32828ebe0
I20260928 13:12:34.881537 29184 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c8fe7eebdc4127b4d583dab44faba6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45243 } }
I20260928 13:12:34.881810 29184 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:34.881858 29184 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c8fe7eebdc4127b4d583dab44faba6, State: Initialized, Role: FOLLOWER
I20260928 13:12:34.882069 29184 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c8fe7eebdc4127b4d583dab44faba6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45243 } }
I20260928 13:12:34.882171 29184 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:34.882251 29184 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:34.882300 29184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:34.883152 29184 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c8fe7eebdc4127b4d583dab44faba6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45243 } }
I20260928 13:12:34.883275 29184 leader_election.cc:302] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2c8fe7eebdc4127b4d583dab44faba6; no voters:
I20260928 13:12:34.883493 29184 leader_election.cc:288] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:34.883674 29189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:34.883798 29184 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:34.883989 29189 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [term 1 LEADER]: Becoming Leader. State: Replica: f2c8fe7eebdc4127b4d583dab44faba6, State: Running, Role: LEADER
I20260928 13:12:34.884135 29189 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c8fe7eebdc4127b4d583dab44faba6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45243 } }
I20260928 13:12:34.884507 29190 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2c8fe7eebdc4127b4d583dab44faba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c8fe7eebdc4127b4d583dab44faba6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45243 } } }
I20260928 13:12:34.884608 29190 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:34.884770 29191 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2c8fe7eebdc4127b4d583dab44faba6. Latest consensus state: current_term: 1 leader_uuid: "f2c8fe7eebdc4127b4d583dab44faba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c8fe7eebdc4127b4d583dab44faba6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45243 } } }
I20260928 13:12:34.884828 29191 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6 [sys.catalog]: This master's current role is: LEADER
W20260928 13:12:34.885897 29204 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:12:34.885957 29204 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:12:34.886004 29198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:34.886180 29198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 13:12:34 dist-test-slave-p1lc krb5kdc[29103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 13:12:34.892547 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:34.879563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37235 (local address 127.0.0.1:45243)
0928 13:12:34.879791 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:34.879803 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:34.879848 (+ 45us) server_negotiation.cc:407] Connection header received
0928 13:12:34.880592 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:34.880603 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:34.880686 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:34.880912 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:34.881660 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:34.883449 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:34.884612 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:34.885356 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:34.885491 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:34.888914 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:34.888919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:34.888919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:34.888933 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:34.890833 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:34.891282 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:34.891286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:34.891286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:34.891380 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:34.891608 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:34.891609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:34.891610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:34.891910 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:34.892089 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:34.892197 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 13:12:34.892348 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":101,"threads_started":1}
I20260928 13:12:34.893146 29198 catalog_manager.cc:1347] Generated new cluster ID: 95e6bcbfd14d4df28b78cd07bde92dd4
I20260928 13:12:34.893203 29198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:34.893560 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:34.908084 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:34.901368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39049 (local address 127.0.0.1:45243)
0928 13:12:34.901417 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:34.901419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:34.901669 (+ 250us) server_negotiation.cc:407] Connection header received
0928 13:12:34.901759 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:34.901761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:34.901772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:34.901845 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:34.902484 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:34.903017 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:34.903756 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:34.903992 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:34.904070 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:34.905426 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:34.905428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:34.905428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:34.905436 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:34.906854 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:34.907263 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:34.907265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:34.907265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:34.907327 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:34.907557 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:34.907558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:34.907558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:34.907679 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:34.907785 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:34.907885 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 13:12:34.907978 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:12:34.922838 29198 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:34.923434 29198 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:34.926162 29198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2c8fe7eebdc4127b4d583dab44faba6: Generated new TSK 0
I20260928 13:12:34.926371 29198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:34.931339 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 29116
2026-09-28T13:12:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:34 dist-test-slave-p1lc krb5kdc[29215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:34 dist-test-slave-p1lc krb5kdc[29215](info): set up 2 sockets
Sep 28 13:12:34 dist-test-slave-p1lc krb5kdc[29215](info): commencing operation
krb5kdc: starting...
W20260928 13:12:35.969060 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:36 dist-test-slave-p1lc krb5kdc[29215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:12:36.099998 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:36.184409 29228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:36.184631 29228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:36.184655 29228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:36.185935 29228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:36.185992 29228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:36.186005 29228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:36.186018 29228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:36.186031 29228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:36.187590 29228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:36.187872 29228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:36.188122 29228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:36.193629 29233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:36.193641 29236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:36.193928 29228 server_base.cc:1034] running on GCE node
W20260928 13:12:36.194977 29234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:36.195263 29228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:36.195544 29228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:36.196725 29228 hybrid_clock.cc:648] HybridClock initialized: now 1790601156196709 us; error 35 us; skew 500 ppm
Sep 28 13:12:36 dist-test-slave-p1lc krb5kdc[29215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:36.199611 29228 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:12:36.200168 29228 webserver.cc:460] Webserver started at http://127.0.0.1:39861/ using document root <none> and password file <none>
I20260928 13:12:36.200428 29228 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:36.200497 29228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:36.200651 29228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:36.201817 29228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "d5c12fcfe73147148b8fc24c87f2b717"
format_stamp: "Formatted at 2026-09-28 13:12:36 on dist-test-slave-p1lc"
server_key: "9506458b3b6d73289884f6b62b2c8460"
server_key_iv: "92d7c5e135f5cb9dca5726138cce374e"
server_key_version: "encryptionkey@0"
I20260928 13:12:36.202186 29228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "d5c12fcfe73147148b8fc24c87f2b717"
format_stamp: "Formatted at 2026-09-28 13:12:36 on dist-test-slave-p1lc"
server_key: "9506458b3b6d73289884f6b62b2c8460"
server_key_iv: "92d7c5e135f5cb9dca5726138cce374e"
server_key_version: "encryptionkey@0"
I20260928 13:12:36.203603 29228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:12:36.204427 29243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:36.204645 29228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:36.204748 29228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "d5c12fcfe73147148b8fc24c87f2b717"
format_stamp: "Formatted at 2026-09-28 13:12:36 on dist-test-slave-p1lc"
server_key: "9506458b3b6d73289884f6b62b2c8460"
server_key_iv: "92d7c5e135f5cb9dca5726138cce374e"
server_key_version: "encryptionkey@0"
I20260928 13:12:36.204835 29228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:36.217519 29228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:36.219678 29228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:36.219856 29228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:36.224452 29228 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37231
I20260928 13:12:36.224489 29295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37231 every 8 connection(s)
I20260928 13:12:36.224915 29228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:36.225498 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 29228
I20260928 13:12:36.225646 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:36.225899 8546 external_mini_cluster.cc:1428] Setting key bf2c6fa111475902b2aedc9c0106ae4a
I20260928 13:12:36.226258 29296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:36.229584 29296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717: Bootstrap starting.
I20260928 13:12:36.230229 29296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:36.230530 29296 log.cc:824] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:36.231372 29296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717: No bootstrap required, opened a new log
I20260928 13:12:36.231972 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:36.226463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40817 (local address 127.0.0.1:37231)
0928 13:12:36.226687 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:36.226695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:36.226721 (+ 26us) server_negotiation.cc:407] Connection header received
0928 13:12:36.227434 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:36.227443 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:36.227494 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:36.227611 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:36.227876 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:36.228789 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:36.229874 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:36.230332 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:36.230484 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:36.231249 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:36.231253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:36.231253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:36.231447 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:36.231495 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:36.231507 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:36.231558 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:36.231642 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 13:12:36.231694 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":112,"threads_started":1}
I20260928 13:12:36.233736 29296 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c12fcfe73147148b8fc24c87f2b717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37231 } }
I20260928 13:12:36.233954 29296 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:36.233963 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:36.234009 29296 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c12fcfe73147148b8fc24c87f2b717, State: Initialized, Role: FOLLOWER
I20260928 13:12:36.234133 29296 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c12fcfe73147148b8fc24c87f2b717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37231 } }
I20260928 13:12:36.234208 29296 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:36.234234 29296 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:36.234270 29296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:36.235121 29296 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c12fcfe73147148b8fc24c87f2b717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37231 } }
I20260928 13:12:36.235248 29296 leader_election.cc:302] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5c12fcfe73147148b8fc24c87f2b717; no voters:
I20260928 13:12:36.235445 29296 leader_election.cc:288] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:36.235504 29305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:36.235663 29305 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [term 1 LEADER]: Becoming Leader. State: Replica: d5c12fcfe73147148b8fc24c87f2b717, State: Running, Role: LEADER
I20260928 13:12:36.235730 29296 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:36.235818 29305 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c12fcfe73147148b8fc24c87f2b717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37231 } }
I20260928 13:12:36.236318 29306 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5c12fcfe73147148b8fc24c87f2b717" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c12fcfe73147148b8fc24c87f2b717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37231 } } }
I20260928 13:12:36.236446 29306 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:36.236634 29306 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5c12fcfe73147148b8fc24c87f2b717. Latest consensus state: current_term: 1 leader_uuid: "d5c12fcfe73147148b8fc24c87f2b717" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c12fcfe73147148b8fc24c87f2b717" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37231 } } }
I20260928 13:12:36.236739 29306 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:36.237119 29315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:36.237634 29315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:36.239392 29299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:36.235845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42107 (local address 127.0.0.1:37231)
0928 13:12:36.235927 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:36.235935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:36.236058 (+ 123us) server_negotiation.cc:407] Connection header received
0928 13:12:36.236161 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:36.236162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:36.236173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:36.236205 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:36.236447 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:36.237245 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:36.238089 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:36.238373 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:36.238451 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:36.239115 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:36.239116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:36.239116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:36.239120 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:36.239136 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:36.239140 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:36.239197 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:36.239291 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 13:12:36.239316 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 13:12:36.239698 29315 catalog_manager.cc:1347] Generated new cluster ID: 3d20cfbef5f14203b050eda6066036b0
I20260928 13:12:36.239847 29315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:36.262542 29315 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:36.263137 29315 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:36.266175 29315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5c12fcfe73147148b8fc24c87f2b717: Generated new TSK 0
I20260928 13:12:36.266391 29315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:36.270087 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 29228
2026-09-28T13:12:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:36 dist-test-slave-p1lc krb5kdc[29327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:36 dist-test-slave-p1lc krb5kdc[29327](info): set up 2 sockets
Sep 28 13:12:36 dist-test-slave-p1lc krb5kdc[29327](info): commencing operation
krb5kdc: starting...
W20260928 13:12:37.317092 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:37 dist-test-slave-p1lc krb5kdc[29327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:12:37.440557 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40227
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:37.523317 29340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:37.523607 29340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:37.523653 29340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:37.524861 29340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:37.524955 29340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:37.524983 29340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:37.525007 29340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:37.525036 29340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:37.526575 29340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:37.526871 29340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:37.527110 29340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:37.532007 29348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:37.532007 29346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:37.532117 29345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:37.532248 29340 server_base.cc:1034] running on GCE node
I20260928 13:12:37.532559 29340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:37.532859 29340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:37.534063 29340 hybrid_clock.cc:648] HybridClock initialized: now 1790601157534024 us; error 59 us; skew 500 ppm
Sep 28 13:12:37 dist-test-slave-p1lc krb5kdc[29327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:37.536729 29340 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:12:37.537310 29340 webserver.cc:460] Webserver started at http://127.0.0.1:41563/ using document root <none> and password file <none>
I20260928 13:12:37.537577 29340 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:37.537647 29340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:37.537797 29340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:37.538858 29340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "adaa4f76bbe341a486f135daedc3d7de"
format_stamp: "Formatted at 2026-09-28 13:12:37 on dist-test-slave-p1lc"
server_key: "342d59a5d40e1b94bada2cc3d5d4c964"
server_key_iv: "f72af008ef09adae57b96d23033007f7"
server_key_version: "encryptionkey@0"
I20260928 13:12:37.539250 29340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "adaa4f76bbe341a486f135daedc3d7de"
format_stamp: "Formatted at 2026-09-28 13:12:37 on dist-test-slave-p1lc"
server_key: "342d59a5d40e1b94bada2cc3d5d4c964"
server_key_iv: "f72af008ef09adae57b96d23033007f7"
server_key_version: "encryptionkey@0"
I20260928 13:12:37.540622 29340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:12:37.541451 29355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:37.541630 29340 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:37.541738 29340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "adaa4f76bbe341a486f135daedc3d7de"
format_stamp: "Formatted at 2026-09-28 13:12:37 on dist-test-slave-p1lc"
server_key: "342d59a5d40e1b94bada2cc3d5d4c964"
server_key_iv: "f72af008ef09adae57b96d23033007f7"
server_key_version: "encryptionkey@0"
I20260928 13:12:37.541826 29340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:37.545305 29340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:37.547286 29340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:37.547448 29340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:37.551854 29340 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40227
I20260928 13:12:37.551998 29407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40227 every 8 connection(s)
I20260928 13:12:37.552271 29340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:37.553364 29408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:37.555512 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 29340
I20260928 13:12:37.555639 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:37.555837 8546 external_mini_cluster.cc:1428] Setting key 1e07738ffe2431be90f006e9fffee34e
I20260928 13:12:37.556149 29408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de: Bootstrap starting.
I20260928 13:12:37.556927 29408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:37.557251 29408 log.cc:824] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:37.558326 29408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de: No bootstrap required, opened a new log
I20260928 13:12:37.558624 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:37.556397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33839 (local address 127.0.0.1:40227)
0928 13:12:37.556612 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:37.556622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:37.556655 (+ 33us) server_negotiation.cc:407] Connection header received
0928 13:12:37.557285 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:37.557293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:37.557320 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:37.557497 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:37.558163 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:37.558167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:37.558167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:37.558179 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:37.558231 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:37.558242 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:37.558342 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:37.558380 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 13:12:37.558421 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260928 13:12:37.559623 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:37.560935 29408 raft_consensus.cc:348] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adaa4f76bbe341a486f135daedc3d7de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260928 13:12:37.561159 29408 raft_consensus.cc:374] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:37.561214 29408 raft_consensus.cc:729] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adaa4f76bbe341a486f135daedc3d7de, State: Initialized, Role: FOLLOWER
I20260928 13:12:37.561367 29408 consensus_queue.cc:260] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adaa4f76bbe341a486f135daedc3d7de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260928 13:12:37.561491 29408 raft_consensus.cc:388] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:37.561565 29408 raft_consensus.cc:482] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:37.561636 29408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:37.562518 29408 raft_consensus.cc:504] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adaa4f76bbe341a486f135daedc3d7de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260928 13:12:37.562644 29408 leader_election.cc:302] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adaa4f76bbe341a486f135daedc3d7de; no voters:
I20260928 13:12:37.562798 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:37.561479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33901 (local address 127.0.0.1:40227)
0928 13:12:37.561525 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:37.561527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:37.561644 (+ 117us) server_negotiation.cc:407] Connection header received
0928 13:12:37.561695 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:37.561696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:37.561702 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:37.561795 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:37.562525 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:37.562526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:37.562526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:37.562531 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:37.562547 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:37.562552 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:37.562622 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:37.562699 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 13:12:37.562713 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:12:37.562836 29408 leader_election.cc:288] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:37.562879 29418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:37.563128 29418 raft_consensus.cc:686] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [term 1 LEADER]: Becoming Leader. State: Replica: adaa4f76bbe341a486f135daedc3d7de, State: Running, Role: LEADER
I20260928 13:12:37.563232 29408 sys_catalog.cc:564] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:37.563263 29418 consensus_queue.cc:237] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adaa4f76bbe341a486f135daedc3d7de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } }
I20260928 13:12:37.563598 29418 sys_catalog.cc:455] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [sys.catalog]: SysCatalogTable state changed. Reason: New leader adaa4f76bbe341a486f135daedc3d7de. Latest consensus state: current_term: 1 leader_uuid: "adaa4f76bbe341a486f135daedc3d7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adaa4f76bbe341a486f135daedc3d7de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } } }
I20260928 13:12:37.563688 29418 sys_catalog.cc:458] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:37.563611 29419 sys_catalog.cc:455] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adaa4f76bbe341a486f135daedc3d7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adaa4f76bbe341a486f135daedc3d7de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40227 } } }
I20260928 13:12:37.563822 29419 sys_catalog.cc:458] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:37.563962 29424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:37.564674 29424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:37.566411 29424 catalog_manager.cc:1347] Generated new cluster ID: 53e6860515854720bdf9874852878d94
I20260928 13:12:37.566489 29424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:37.572230 29424 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:37.572609 29424 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:37.577455 29424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adaa4f76bbe341a486f135daedc3d7de: Generated new TSK 0
I20260928 13:12:37.577672 29424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:37.579779 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 29340
2026-09-28T13:12:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:37 dist-test-slave-p1lc krb5kdc[29439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:37 dist-test-slave-p1lc krb5kdc[29439](info): set up 2 sockets
Sep 28 13:12:37 dist-test-slave-p1lc krb5kdc[29439](info): commencing operation
krb5kdc: starting...
W20260928 13:12:38.637562 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:38 dist-test-slave-p1lc krb5kdc[29439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:12:38.764353 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:38.846244 29452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:38.846505 29452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:38.846542 29452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:38.847709 29452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:38.847785 29452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:38.847810 29452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:38.847831 29452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:38.847860 29452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:38.849365 29452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:38.849650 29452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:38.849891 29452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:38.854902 29460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:38.854996 29458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:38.855022 29452 server_base.cc:1034] running on GCE node
W20260928 13:12:38.854902 29457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:38.855405 29452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:38.855718 29452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:38.856913 29452 hybrid_clock.cc:648] HybridClock initialized: now 1790601158856892 us; error 37 us; skew 500 ppm
Sep 28 13:12:38 dist-test-slave-p1lc krb5kdc[29439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:38.866197 29452 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:12:38.866894 29452 webserver.cc:460] Webserver started at http://127.0.0.1:40561/ using document root <none> and password file <none>
I20260928 13:12:38.867218 29452 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:38.867290 29452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:38.867441 29452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:38.868762 29452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "736db8f17aa64656a2cff118c1086000"
format_stamp: "Formatted at 2026-09-28 13:12:38 on dist-test-slave-p1lc"
server_key: "1cf877af04b233c64dfa91438f31934e"
server_key_iv: "d2fd6dbb540d13d8afb1e27eb23bf150"
server_key_version: "encryptionkey@0"
I20260928 13:12:38.869225 29452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "736db8f17aa64656a2cff118c1086000"
format_stamp: "Formatted at 2026-09-28 13:12:38 on dist-test-slave-p1lc"
server_key: "1cf877af04b233c64dfa91438f31934e"
server_key_iv: "d2fd6dbb540d13d8afb1e27eb23bf150"
server_key_version: "encryptionkey@0"
I20260928 13:12:38.870769 29452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:12:38.871739 29467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:38.871948 29452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:38.872072 29452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "736db8f17aa64656a2cff118c1086000"
format_stamp: "Formatted at 2026-09-28 13:12:38 on dist-test-slave-p1lc"
server_key: "1cf877af04b233c64dfa91438f31934e"
server_key_iv: "d2fd6dbb540d13d8afb1e27eb23bf150"
server_key_version: "encryptionkey@0"
I20260928 13:12:38.872157 29452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:38.884472 29452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:38.887326 29452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:38.887511 29452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:38.891877 29452 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40167
I20260928 13:12:38.891953 29519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40167 every 8 connection(s)
I20260928 13:12:38.892339 29452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:38.893468 29520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:38.895924 29520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000: Bootstrap starting.
I20260928 13:12:38.896536 29520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:38.896834 29520 log.cc:824] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:38.897750 29520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000: No bootstrap required, opened a new log
I20260928 13:12:38.899535 29520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736db8f17aa64656a2cff118c1086000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } }
I20260928 13:12:38.899695 29520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:38.899720 29520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736db8f17aa64656a2cff118c1086000, State: Initialized, Role: FOLLOWER
I20260928 13:12:38.899887 29520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736db8f17aa64656a2cff118c1086000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } }
I20260928 13:12:38.899945 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 29452
I20260928 13:12:38.899960 29520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:38.900017 29520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:38.900061 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:38.900070 29520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:38.900251 8546 external_mini_cluster.cc:1428] Setting key 36d25d852e9819ec67d0bb69a51bb964
I20260928 13:12:38.900907 29520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736db8f17aa64656a2cff118c1086000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } }
I20260928 13:12:38.901104 29520 leader_election.cc:302] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 736db8f17aa64656a2cff118c1086000; no voters:
I20260928 13:12:38.901273 29520 leader_election.cc:288] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:38.901340 29525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:38.901517 29525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [term 1 LEADER]: Becoming Leader. State: Replica: 736db8f17aa64656a2cff118c1086000, State: Running, Role: LEADER
I20260928 13:12:38.901551 29520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:38.901633 29525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736db8f17aa64656a2cff118c1086000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } }
I20260928 13:12:38.902040 29526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "736db8f17aa64656a2cff118c1086000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736db8f17aa64656a2cff118c1086000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } } }
I20260928 13:12:38.902030 29525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 736db8f17aa64656a2cff118c1086000. Latest consensus state: current_term: 1 leader_uuid: "736db8f17aa64656a2cff118c1086000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "736db8f17aa64656a2cff118c1086000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40167 } } }
I20260928 13:12:38.902180 29526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:38.902185 29525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:38.903174 29529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:38.903729 29529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 13:12:38 dist-test-slave-p1lc krb5kdc[29439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 13:12:38.908608 29529 catalog_manager.cc:1347] Generated new cluster ID: 46b004d8fc484f99888e45f7e462e3c3
I20260928 13:12:38.908679 29529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:38.912844 29524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:38.900823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:35227 (local address 127.0.0.1:40167)
0928 13:12:38.901082 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:38.901093 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:38.901136 (+ 43us) server_negotiation.cc:407] Connection header received
0928 13:12:38.901274 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:38.901282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:38.901387 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:38.902370 (+ 983us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:38.902493 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:38.904203 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:38.905229 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:38.905817 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:38.905948 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:38.908614 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:38.908618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:38.908618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:38.908633 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:38.910730 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:38.911211 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:38.911214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:38.911215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:38.911334 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:38.911555 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:38.911555 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:38.911556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:38.911860 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:38.912421 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:38.912506 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 13:12:38.912610 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":141,"threads_started":1}
I20260928 13:12:38.914095 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:38.923378 29524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:38.915667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:50665 (local address 127.0.0.1:40167)
0928 13:12:38.915710 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:38.915712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:38.915851 (+ 139us) server_negotiation.cc:407] Connection header received
0928 13:12:38.915928 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:38.915929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:38.915941 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:38.915996 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:38.916660 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:38.917442 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:38.918555 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:38.918872 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:38.918946 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:38.920462 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:38.920464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:38.920464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:38.920472 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:38.922114 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:38.922557 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:38.922557 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:38.922558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:38.922615 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:38.922845 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:38.922845 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:38.922846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:38.922975 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:38.923072 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:38.923223 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 13:12:38.923283 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:12:38.925233 29529 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:38.925752 29529 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:38.928556 29529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 736db8f17aa64656a2cff118c1086000: Generated new TSK 0
I20260928 13:12:38.928766 29529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:38.933230 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 29452
W20260928 13:12:38.937827 29437 connection.cc:535] client connection to 127.0.0.1:40167 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40167): Connection reset by peer (error 104)
2026-09-28T13:12:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:38 dist-test-slave-p1lc krb5kdc[29551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:38 dist-test-slave-p1lc krb5kdc[29551](info): set up 2 sockets
Sep 28 13:12:38 dist-test-slave-p1lc krb5kdc[29551](info): commencing operation
krb5kdc: starting...
W20260928 13:12:39.985288 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:40 dist-test-slave-p1lc krb5kdc[29551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:12:40.112381 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:40.196208 29564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:40.196460 29564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:40.196496 29564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:40.197697 29564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:40.197773 29564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:40.197798 29564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:40.197818 29564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:40.197847 29564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:40.199339 29564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:40.199625 29564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:40.199860 29564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:40.204988 29570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:40.205024 29569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:40.205118 29564 server_base.cc:1034] running on GCE node
W20260928 13:12:40.205060 29572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:40.205428 29564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:40.205729 29564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:40.206905 29564 hybrid_clock.cc:648] HybridClock initialized: now 1790601160206874 us; error 60 us; skew 500 ppm
Sep 28 13:12:40 dist-test-slave-p1lc krb5kdc[29551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:40.210053 29564 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:12:40.210592 29564 webserver.cc:460] Webserver started at http://127.0.0.1:38029/ using document root <none> and password file <none>
I20260928 13:12:40.210911 29564 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:40.210975 29564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:40.211129 29564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:40.212370 29564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "195789a5e3084caa8bd48dfb1f879c9e"
format_stamp: "Formatted at 2026-09-28 13:12:40 on dist-test-slave-p1lc"
server_key: "2529006d7ae830a82bf4557e207e9b67"
server_key_iv: "6fe1cafd59f505a0f43edff14f98addc"
server_key_version: "encryptionkey@0"
I20260928 13:12:40.212682 29564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "195789a5e3084caa8bd48dfb1f879c9e"
format_stamp: "Formatted at 2026-09-28 13:12:40 on dist-test-slave-p1lc"
server_key: "2529006d7ae830a82bf4557e207e9b67"
server_key_iv: "6fe1cafd59f505a0f43edff14f98addc"
server_key_version: "encryptionkey@0"
I20260928 13:12:40.214082 29564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:40.214885 29579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:40.215095 29564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:40.215186 29564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "195789a5e3084caa8bd48dfb1f879c9e"
format_stamp: "Formatted at 2026-09-28 13:12:40 on dist-test-slave-p1lc"
server_key: "2529006d7ae830a82bf4557e207e9b67"
server_key_iv: "6fe1cafd59f505a0f43edff14f98addc"
server_key_version: "encryptionkey@0"
I20260928 13:12:40.215246 29564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:40.224695 29564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:40.226814 29564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:40.226969 29564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:40.230958 29564 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45131
I20260928 13:12:40.230989 29631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45131 every 8 connection(s)
I20260928 13:12:40.231362 29564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:40.232367 29632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:40.234712 29632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e: Bootstrap starting.
I20260928 13:12:40.235299 29632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:40.235606 29632 log.cc:824] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:40.236370 29632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e: No bootstrap required, opened a new log
I20260928 13:12:40.237802 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 29564
I20260928 13:12:40.237939 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:40.238168 8546 external_mini_cluster.cc:1428] Setting key 0f032a4750c21a8201de7f540a54b14d
I20260928 13:12:40.238174 29632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195789a5e3084caa8bd48dfb1f879c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260928 13:12:40.238384 29632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:40.238433 29632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195789a5e3084caa8bd48dfb1f879c9e, State: Initialized, Role: FOLLOWER
I20260928 13:12:40.238585 29632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195789a5e3084caa8bd48dfb1f879c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260928 13:12:40.238682 29632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:40.238751 29632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:40.238850 29632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:40.239725 29632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195789a5e3084caa8bd48dfb1f879c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260928 13:12:40.239848 29632 leader_election.cc:302] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 195789a5e3084caa8bd48dfb1f879c9e; no voters:
I20260928 13:12:40.240016 29632 leader_election.cc:288] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:40.240056 29637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:40.240165 29637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [term 1 LEADER]: Becoming Leader. State: Replica: 195789a5e3084caa8bd48dfb1f879c9e, State: Running, Role: LEADER
I20260928 13:12:40.240289 29632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:40.240302 29637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195789a5e3084caa8bd48dfb1f879c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } }
I20260928 13:12:40.240775 29637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 195789a5e3084caa8bd48dfb1f879c9e. Latest consensus state: current_term: 1 leader_uuid: "195789a5e3084caa8bd48dfb1f879c9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195789a5e3084caa8bd48dfb1f879c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } } }
I20260928 13:12:40.241070 29637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:40.240971 29638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "195789a5e3084caa8bd48dfb1f879c9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195789a5e3084caa8bd48dfb1f879c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45131 } } }
I20260928 13:12:40.241647 29638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e [sys.catalog]: This master's current role is: LEADER
W20260928 13:12:40.242084 29653 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:12:40.242133 29653 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:12:40.242209 29646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:40.242419 29646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:40.243981 29646 catalog_manager.cc:1347] Generated new cluster ID: 9c4261f93bbc4dc083114bf8a7bdeb47
I20260928 13:12:40.244041 29646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:40.245091 29636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:40.238771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:57973 (local address 127.0.0.1:45131)
0928 13:12:40.239108 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:40.239118 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:40.239159 (+ 41us) server_negotiation.cc:407] Connection header received
0928 13:12:40.239277 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:40.239282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:40.239330 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:40.239487 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:40.239704 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:40.241211 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:40.242480 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:40.243541 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:40.243673 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:40.244447 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:40.244450 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:40.244451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:40.244646 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:40.244679 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:40.244690 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:40.244782 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:40.244864 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 13:12:40.244921 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":117,"threads_started":1}
I20260928 13:12:40.249680 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:40.256330 29636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:40.253229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:40973 (local address 127.0.0.1:45131)
0928 13:12:40.253295 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:40.253297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:40.253466 (+ 169us) server_negotiation.cc:407] Connection header received
0928 13:12:40.253558 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:40.253559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:40.253573 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:40.253628 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:40.253796 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:40.254313 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:40.255063 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:40.255272 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:40.255349 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:40.256047 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:40.256048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:40.256048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:40.256053 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:40.256068 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:40.256077 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:40.256140 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:40.256207 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 13:12:40.256219 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":40}
I20260928 13:12:40.257520 29646 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:40.258118 29646 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:40.271137 29646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 195789a5e3084caa8bd48dfb1f879c9e: Generated new TSK 0
I20260928 13:12:40.271349 29646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:40.278262 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 29564
W20260928 13:12:40.282781 29549 connection.cc:535] client connection to 127.0.0.1:45131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45131): Connection reset by peer (error 104)
2026-09-28T13:12:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:40 dist-test-slave-p1lc krb5kdc[29663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:40 dist-test-slave-p1lc krb5kdc[29663](info): set up 2 sockets
Sep 28 13:12:40 dist-test-slave-p1lc krb5kdc[29663](info): commencing operation
krb5kdc: starting...
W20260928 13:12:41.333715 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:41 dist-test-slave-p1lc krb5kdc[29663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:12:41.463616 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:41.547181 29676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:41.547408 29676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:41.547434 29676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:41.548669 29676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:41.548724 29676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:41.548738 29676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:41.548751 29676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:41.548810 29676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:41.550391 29676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:41.550671 29676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:41.550917 29676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:41.556329 29684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:41.556350 29681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:41.556546 29682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:41.556416 29676 server_base.cc:1034] running on GCE node
I20260928 13:12:41.556828 29676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:41.557201 29676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:41.558395 29676 hybrid_clock.cc:648] HybridClock initialized: now 1790601161558365 us; error 52 us; skew 500 ppm
Sep 28 13:12:41 dist-test-slave-p1lc krb5kdc[29663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:41.561127 29676 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:12:41.561684 29676 webserver.cc:460] Webserver started at http://127.0.0.1:37557/ using document root <none> and password file <none>
I20260928 13:12:41.561942 29676 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:41.562014 29676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:41.562170 29676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:41.563319 29676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "7fbff70854f44f5a98c0ad47cf710bcf"
format_stamp: "Formatted at 2026-09-28 13:12:41 on dist-test-slave-p1lc"
server_key: "0ba857a63e246401f1e416fe5807fd69"
server_key_iv: "fa52927eb3d5e802c711c038603c7939"
server_key_version: "encryptionkey@0"
I20260928 13:12:41.563702 29676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "7fbff70854f44f5a98c0ad47cf710bcf"
format_stamp: "Formatted at 2026-09-28 13:12:41 on dist-test-slave-p1lc"
server_key: "0ba857a63e246401f1e416fe5807fd69"
server_key_iv: "fa52927eb3d5e802c711c038603c7939"
server_key_version: "encryptionkey@0"
I20260928 13:12:41.565224 29676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 13:12:41.566205 29691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:41.566442 29676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:41.566568 29676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "7fbff70854f44f5a98c0ad47cf710bcf"
format_stamp: "Formatted at 2026-09-28 13:12:41 on dist-test-slave-p1lc"
server_key: "0ba857a63e246401f1e416fe5807fd69"
server_key_iv: "fa52927eb3d5e802c711c038603c7939"
server_key_version: "encryptionkey@0"
I20260928 13:12:41.566712 29676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:41.570349 29676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:41.572531 29676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:41.572708 29676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:41.577108 29676 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33723
I20260928 13:12:41.577158 29743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33723 every 8 connection(s)
I20260928 13:12:41.577538 29676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:41.578543 29744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:41.578773 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 29676
I20260928 13:12:41.578881 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:41.579095 8546 external_mini_cluster.cc:1428] Setting key 21827d8c140e4e2bdbce3cd4722dd743
I20260928 13:12:41.581533 29744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf: Bootstrap starting.
I20260928 13:12:41.581791 29747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:41.579660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:43673 (local address 127.0.0.1:33723)
0928 13:12:41.579876 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:41.579884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:41.579910 (+ 26us) server_negotiation.cc:407] Connection header received
0928 13:12:41.580377 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:41.580384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:41.580415 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:41.580580 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:41.581327 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:41.581331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:41.581331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:41.581335 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:41.581387 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:41.581398 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:41.581476 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:41.581556 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 13:12:41.581602 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260928 13:12:41.582367 29744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:41.582736 29744 log.cc:824] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:41.583112 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:41.583583 29744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf: No bootstrap required, opened a new log
I20260928 13:12:41.586195 29747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:41.584778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:54125 (local address 127.0.0.1:33723)
0928 13:12:41.584838 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:41.584840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:41.585107 (+ 267us) server_negotiation.cc:407] Connection header received
0928 13:12:41.585148 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:41.585149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:41.585154 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:41.585240 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:41.585875 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:41.585877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:41.585878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:41.585883 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:41.585902 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:41.585907 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:41.585987 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:41.586117 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 13:12:41.586132 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 13:12:41.586380 29744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbff70854f44f5a98c0ad47cf710bcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } }
I20260928 13:12:41.586572 29744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:41.586621 29744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbff70854f44f5a98c0ad47cf710bcf, State: Initialized, Role: FOLLOWER
I20260928 13:12:41.586747 29744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbff70854f44f5a98c0ad47cf710bcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } }
I20260928 13:12:41.586854 29744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:41.586900 29744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:41.586953 29744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:41.587550 29744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbff70854f44f5a98c0ad47cf710bcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } }
I20260928 13:12:41.587677 29744 leader_election.cc:302] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fbff70854f44f5a98c0ad47cf710bcf; no voters:
I20260928 13:12:41.587872 29744 leader_election.cc:288] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:41.587952 29754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:41.588128 29754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [term 1 LEADER]: Becoming Leader. State: Replica: 7fbff70854f44f5a98c0ad47cf710bcf, State: Running, Role: LEADER
I20260928 13:12:41.588135 29744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:41.588241 29754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbff70854f44f5a98c0ad47cf710bcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } }
I20260928 13:12:41.588656 29755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fbff70854f44f5a98c0ad47cf710bcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbff70854f44f5a98c0ad47cf710bcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } } }
I20260928 13:12:41.588858 29755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:41.588665 29756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fbff70854f44f5a98c0ad47cf710bcf. Latest consensus state: current_term: 1 leader_uuid: "7fbff70854f44f5a98c0ad47cf710bcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fbff70854f44f5a98c0ad47cf710bcf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33723 } } }
I20260928 13:12:41.589401 29756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf [sys.catalog]: This master's current role is: LEADER
W20260928 13:12:41.590067 29770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:12:41.590114 29770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:12:41.590183 29763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:41.590365 29763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:41.591859 29763 catalog_manager.cc:1347] Generated new cluster ID: b2e4c7be44ec4cf8a0070b01885e28e0
I20260928 13:12:41.591910 29763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:41.597239 29763 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:41.597414 29763 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:41.602398 29763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fbff70854f44f5a98c0ad47cf710bcf: Generated new TSK 0
I20260928 13:12:41.602609 29763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:41.607071 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 29676
2026-09-28T13:12:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:41 dist-test-slave-p1lc krb5kdc[29775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:41 dist-test-slave-p1lc krb5kdc[29775](info): set up 2 sockets
Sep 28 13:12:41 dist-test-slave-p1lc krb5kdc[29775](info): commencing operation
krb5kdc: starting...
W20260928 13:12:42.705233 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:42 dist-test-slave-p1lc krb5kdc[29775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:12:42.832825 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:42.918107 29788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:42.918383 29788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:42.918424 29788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:42.919708 29788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:42.919800 29788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:42.919826 29788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:42.919847 29788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:42.919876 29788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:42.921561 29788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:42.921883 29788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:42.922142 29788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:42.927551 29796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:42.927569 29794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:42.927546 29793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:42.927932 29788 server_base.cc:1034] running on GCE node
I20260928 13:12:42.928228 29788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:42.928503 29788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:42.929652 29788 hybrid_clock.cc:648] HybridClock initialized: now 1790601162929637 us; error 31 us; skew 500 ppm
Sep 28 13:12:42 dist-test-slave-p1lc krb5kdc[29775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:42.932268 29788 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:12:42.932753 29788 webserver.cc:460] Webserver started at http://127.0.0.1:35789/ using document root <none> and password file <none>
I20260928 13:12:42.933050 29788 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:42.933110 29788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:42.933250 29788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:42.934398 29788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "63f6776cd2d5422eba6239d8668a1f8e"
format_stamp: "Formatted at 2026-09-28 13:12:42 on dist-test-slave-p1lc"
server_key: "d70770bbdde0e6a17db23ef20a8bac23"
server_key_iv: "00532b0fa902e122bc0a49a6eb6e6f7d"
server_key_version: "encryptionkey@0"
I20260928 13:12:42.934778 29788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "63f6776cd2d5422eba6239d8668a1f8e"
format_stamp: "Formatted at 2026-09-28 13:12:42 on dist-test-slave-p1lc"
server_key: "d70770bbdde0e6a17db23ef20a8bac23"
server_key_iv: "00532b0fa902e122bc0a49a6eb6e6f7d"
server_key_version: "encryptionkey@0"
I20260928 13:12:42.936290 29788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:42.937201 29803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:42.937414 29788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:42.937497 29788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "63f6776cd2d5422eba6239d8668a1f8e"
format_stamp: "Formatted at 2026-09-28 13:12:42 on dist-test-slave-p1lc"
server_key: "d70770bbdde0e6a17db23ef20a8bac23"
server_key_iv: "00532b0fa902e122bc0a49a6eb6e6f7d"
server_key_version: "encryptionkey@0"
I20260928 13:12:42.937618 29788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:42.951642 29788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:42.954005 29788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:42.954164 29788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:42.958295 29788 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40849
I20260928 13:12:42.958349 29855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40849 every 8 connection(s)
I20260928 13:12:42.958685 29788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:42.959748 29856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:42.962134 29856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e: Bootstrap starting.
I20260928 13:12:42.962695 29856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:42.962931 29856 log.cc:824] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:42.963671 29856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e: No bootstrap required, opened a new log
I20260928 13:12:42.965466 29856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f6776cd2d5422eba6239d8668a1f8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260928 13:12:42.965611 29856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:42.965633 29856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63f6776cd2d5422eba6239d8668a1f8e, State: Initialized, Role: FOLLOWER
I20260928 13:12:42.965740 29856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f6776cd2d5422eba6239d8668a1f8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260928 13:12:42.965821 29856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:42.965858 29856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:42.965926 29856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:42.966507 29856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f6776cd2d5422eba6239d8668a1f8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260928 13:12:42.966627 29856 leader_election.cc:302] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63f6776cd2d5422eba6239d8668a1f8e; no voters:
I20260928 13:12:42.966827 29856 leader_election.cc:288] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:42.966884 29859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:42.967206 29859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [term 1 LEADER]: Becoming Leader. State: Replica: 63f6776cd2d5422eba6239d8668a1f8e, State: Running, Role: LEADER
I20260928 13:12:42.967314 29856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:42.967355 29859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f6776cd2d5422eba6239d8668a1f8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260928 13:12:42.968109 29861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63f6776cd2d5422eba6239d8668a1f8e. Latest consensus state: current_term: 1 leader_uuid: "63f6776cd2d5422eba6239d8668a1f8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f6776cd2d5422eba6239d8668a1f8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } } }
I20260928 13:12:42.968111 29860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63f6776cd2d5422eba6239d8668a1f8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63f6776cd2d5422eba6239d8668a1f8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } } }
I20260928 13:12:42.968261 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 29788
I20260928 13:12:42.968283 29860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:42.968228 29861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:42.968392 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:42.968580 29869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:42.968607 8546 external_mini_cluster.cc:1428] Setting key fd2d5a91f7cacc8b579814d820a18609
I20260928 13:12:42.969178 29869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:42.971112 29869 catalog_manager.cc:1347] Generated new cluster ID: ef22d5154e124ef293a060e86b3623a6
I20260928 13:12:42.971171 29869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:42.977018 29869 catalog_manager.cc:1370] Generated new certificate authority record
Sep 28 13:12:42 dist-test-slave-p1lc krb5kdc[29775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 13:12:42.977720 29869 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:42.981309 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:42.969110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:33865 (local address 127.0.0.1:40849)
0928 13:12:42.969289 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:42.969298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:42.969991 (+ 693us) server_negotiation.cc:407] Connection header received
0928 13:12:42.970151 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:42.970159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:42.970232 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:42.971257 (+ 1025us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:42.971361 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:42.972752 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:42.973874 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:42.974536 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:42.974982 (+ 446us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:42.977837 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:42.977841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:42.977842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:42.977855 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:42.979542 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:42.980022 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:42.980026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:42.980026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:42.980125 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:42.980369 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:42.980370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:42.980371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:42.980612 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:42.980777 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:42.980933 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 13:12:42.981044 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260928 13:12:42.982295 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:42.984531 29869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63f6776cd2d5422eba6239d8668a1f8e: Generated new TSK 0
I20260928 13:12:42.984719 29869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:42.990671 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:42.983718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:56513 (local address 127.0.0.1:40849)
0928 13:12:42.983775 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:42.983777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:42.983905 (+ 128us) server_negotiation.cc:407] Connection header received
0928 13:12:42.983981 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:42.983982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:42.983992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:42.984048 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:42.984763 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:42.985365 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:42.986134 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:42.986377 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:42.986458 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:42.987927 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:42.987929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:42.987929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:42.987937 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:42.989456 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:42.989848 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:42.989849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:42.989850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:42.989915 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:42.990149 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:42.990150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:42.990150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:42.990264 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:42.990378 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:42.990499 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 13:12:42.990567 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:12:42.992772 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 29788
W20260928 13:12:42.997300 29773 connection.cc:535] client connection to 127.0.0.1:40849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40849): Connection reset by peer (error 104)
2026-09-28T13:12:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:43 dist-test-slave-p1lc krb5kdc[29887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:43 dist-test-slave-p1lc krb5kdc[29887](info): set up 2 sockets
Sep 28 13:12:43 dist-test-slave-p1lc krb5kdc[29887](info): commencing operation
krb5kdc: starting...
W20260928 13:12:44.029285 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:44 dist-test-slave-p1lc krb5kdc[29887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:12:44.155817 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:44.239122 29900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:44.239344 29900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:44.239367 29900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:44.240484 29900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:44.240522 29900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:44.240536 29900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:44.240548 29900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:44.240561 29900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:44.242106 29900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36131
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:44.242311 29900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:44.242482 29900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:44.247449 29908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:44.247603 29900 server_base.cc:1034] running on GCE node
W20260928 13:12:44.247473 29906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:44.247691 29905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:44.247987 29900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:44.248245 29900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:44.249384 29900 hybrid_clock.cc:648] HybridClock initialized: now 1790601164249368 us; error 28 us; skew 500 ppm
Sep 28 13:12:44 dist-test-slave-p1lc krb5kdc[29887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:44.256850 29900 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:12:44.257355 29900 webserver.cc:460] Webserver started at http://127.0.0.1:45961/ using document root <none> and password file <none>
I20260928 13:12:44.257566 29900 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:44.257609 29900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:44.257702 29900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:44.258782 29900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "f46b51fbc6f04264b2bffac7cca22eb1"
format_stamp: "Formatted at 2026-09-28 13:12:44 on dist-test-slave-p1lc"
server_key: "616681e0e12c8cd9ea2f9d461382251d"
server_key_iv: "870b2c15df4f0f9054cbdcaa3934fb88"
server_key_version: "encryptionkey@0"
I20260928 13:12:44.259090 29900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "f46b51fbc6f04264b2bffac7cca22eb1"
format_stamp: "Formatted at 2026-09-28 13:12:44 on dist-test-slave-p1lc"
server_key: "616681e0e12c8cd9ea2f9d461382251d"
server_key_iv: "870b2c15df4f0f9054cbdcaa3934fb88"
server_key_version: "encryptionkey@0"
I20260928 13:12:44.260461 29900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:44.261341 29915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:44.261536 29900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 13:12:44.261649 29900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "f46b51fbc6f04264b2bffac7cca22eb1"
format_stamp: "Formatted at 2026-09-28 13:12:44 on dist-test-slave-p1lc"
server_key: "616681e0e12c8cd9ea2f9d461382251d"
server_key_iv: "870b2c15df4f0f9054cbdcaa3934fb88"
server_key_version: "encryptionkey@0"
I20260928 13:12:44.261729 29900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:44.279832 29900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:44.282585 29900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:44.282768 29900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:44.287066 29900 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36131
I20260928 13:12:44.287289 29967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36131 every 8 connection(s)
I20260928 13:12:44.287524 29900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:44.288463 29968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:44.290923 29968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1: Bootstrap starting.
I20260928 13:12:44.291195 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 29900
I20260928 13:12:44.291338 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:44.291596 8546 external_mini_cluster.cc:1428] Setting key 4b4cabcacb06a6f3c005b76c39a80f37
I20260928 13:12:44.291749 29968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:44.292111 29968 log.cc:824] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:44.292964 29968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1: No bootstrap required, opened a new log
I20260928 13:12:44.295230 29968 raft_consensus.cc:348] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46b51fbc6f04264b2bffac7cca22eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36131 } }
I20260928 13:12:44.295418 29968 raft_consensus.cc:374] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:44.295470 29968 raft_consensus.cc:729] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46b51fbc6f04264b2bffac7cca22eb1, State: Initialized, Role: FOLLOWER
I20260928 13:12:44.295594 29968 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46b51fbc6f04264b2bffac7cca22eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36131 } }
I20260928 13:12:44.295667 29968 raft_consensus.cc:388] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:44.295696 29968 raft_consensus.cc:482] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:44.295740 29968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:44.296366 29968 raft_consensus.cc:504] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46b51fbc6f04264b2bffac7cca22eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36131 } }
I20260928 13:12:44.296504 29968 leader_election.cc:302] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f46b51fbc6f04264b2bffac7cca22eb1; no voters:
I20260928 13:12:44.296705 29968 leader_election.cc:288] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:44.296762 29973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:44.297019 29968 sys_catalog.cc:564] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:44.297016 29973 raft_consensus.cc:686] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [term 1 LEADER]: Becoming Leader. State: Replica: f46b51fbc6f04264b2bffac7cca22eb1, State: Running, Role: LEADER
I20260928 13:12:44.297358 29973 consensus_queue.cc:237] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46b51fbc6f04264b2bffac7cca22eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36131 } }
I20260928 13:12:44.298210 29975 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f46b51fbc6f04264b2bffac7cca22eb1. Latest consensus state: current_term: 1 leader_uuid: "f46b51fbc6f04264b2bffac7cca22eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46b51fbc6f04264b2bffac7cca22eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36131 } } }
I20260928 13:12:44.298210 29974 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f46b51fbc6f04264b2bffac7cca22eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46b51fbc6f04264b2bffac7cca22eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36131 } } }
I20260928 13:12:44.298374 29975 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:44.298396 29974 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:44.298574 29971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:44.292183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:57891 (local address 127.0.0.1:36131)
0928 13:12:44.292383 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:44.292394 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:44.292437 (+ 43us) server_negotiation.cc:407] Connection header received
0928 13:12:44.293159 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:44.293167 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:44.293234 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:44.293356 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:44.293683 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:44.294674 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:44.295872 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:44.296358 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:44.296452 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:44.297394 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:44.297399 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:44.297399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:44.297728 (+ 329us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:44.297775 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:44.297792 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:44.298135 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:44.298182 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 13:12:44.298214 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260928 13:12:44.298776 29982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:44.299362 29982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:44.299764 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:44.301224 29982 catalog_manager.cc:1347] Generated new cluster ID: a2ad83fe050f4c9c8eb366963604d100
I20260928 13:12:44.301288 29982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:44.308817 29971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:44.305025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:50131 (local address 127.0.0.1:36131)
0928 13:12:44.305098 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:44.305101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:44.305112 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:12:44.305142 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:44.305144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:44.305155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:44.305240 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:44.305449 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:44.306260 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:44.307334 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:44.307678 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:44.307758 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:44.308542 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:44.308544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:44.308544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:44.308550 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:44.308570 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:44.308575 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:44.308640 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:44.308713 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 13:12:44.308724 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 13:12:44.318154 29982 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:44.318708 29982 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:44.324396 29982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f46b51fbc6f04264b2bffac7cca22eb1: Generated new TSK 0
I20260928 13:12:44.324625 29982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:44.329174 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 29900
W20260928 13:12:44.333730 29885 connection.cc:535] client connection to 127.0.0.1:36131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36131): Connection reset by peer (error 104)
2026-09-28T13:12:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:44 dist-test-slave-p1lc krb5kdc[29999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:44 dist-test-slave-p1lc krb5kdc[29999](info): set up 2 sockets
Sep 28 13:12:44 dist-test-slave-p1lc krb5kdc[29999](info): commencing operation
krb5kdc: starting...
W20260928 13:12:45.377174 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:45 dist-test-slave-p1lc krb5kdc[29999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 13:12:45.504825 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:45.588109 30012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:45.588335 30012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:45.588359 30012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:45.589680 30012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:45.589736 30012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:45.589751 30012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:45.589764 30012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:45.589776 30012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:45.591315 30012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:45.591581 30012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:45.591825 30012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:45.596994 30017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:45.596997 30018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:45.596984 30020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:45.597314 30012 server_base.cc:1034] running on GCE node
I20260928 13:12:45.597604 30012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:45.597911 30012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:45.599107 30012 hybrid_clock.cc:648] HybridClock initialized: now 1790601165599075 us; error 54 us; skew 500 ppm
Sep 28 13:12:45 dist-test-slave-p1lc krb5kdc[29999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:45.601991 30012 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 13:12:45.602452 30012 webserver.cc:460] Webserver started at http://127.0.0.1:34969/ using document root <none> and password file <none>
I20260928 13:12:45.602705 30012 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:45.602775 30012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:45.602931 30012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:45.604028 30012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "3277072a21434d7e9021d06556de694c"
format_stamp: "Formatted at 2026-09-28 13:12:45 on dist-test-slave-p1lc"
server_key: "c7245acde7bd4be919b0c846d0d220ce"
server_key_iv: "79d14f06f7a95eab7aab159273c3e967"
server_key_version: "encryptionkey@0"
I20260928 13:12:45.604413 30012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "3277072a21434d7e9021d06556de694c"
format_stamp: "Formatted at 2026-09-28 13:12:45 on dist-test-slave-p1lc"
server_key: "c7245acde7bd4be919b0c846d0d220ce"
server_key_iv: "79d14f06f7a95eab7aab159273c3e967"
server_key_version: "encryptionkey@0"
I20260928 13:12:45.605875 30012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:12:45.606760 30027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:45.607002 30012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:12:45.607100 30012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "3277072a21434d7e9021d06556de694c"
format_stamp: "Formatted at 2026-09-28 13:12:45 on dist-test-slave-p1lc"
server_key: "c7245acde7bd4be919b0c846d0d220ce"
server_key_iv: "79d14f06f7a95eab7aab159273c3e967"
server_key_version: "encryptionkey@0"
I20260928 13:12:45.607187 30012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:45.610646 30012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:45.613211 30012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:45.613386 30012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:45.617709 30012 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35297
I20260928 13:12:45.617740 30079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35297 every 8 connection(s)
I20260928 13:12:45.618106 30012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:45.619189 30080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:45.620313 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 30012
I20260928 13:12:45.620447 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:45.620672 8546 external_mini_cluster.cc:1428] Setting key ed0e70e7cd9761c3339ae26cfaf80ae4
I20260928 13:12:45.622269 30080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c: Bootstrap starting.
I20260928 13:12:45.623333 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:45.621293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:46273 (local address 127.0.0.1:35297)
0928 13:12:45.621492 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:45.621501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:45.621558 (+ 57us) server_negotiation.cc:407] Connection header received
0928 13:12:45.622002 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:45.622008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:45.622035 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:45.622158 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:45.622864 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:45.622867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:45.622867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:45.622870 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:45.622906 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:45.622917 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:45.622987 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:45.623063 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 13:12:45.623132 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260928 13:12:45.623339 30080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:45.623742 30080 log.cc:824] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:45.624608 30080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c: No bootstrap required, opened a new log
I20260928 13:12:45.624804 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:45.627594 30080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3277072a21434d7e9021d06556de694c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260928 13:12:45.627763 30083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:45.626491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.181:49345 (local address 127.0.0.1:35297)
0928 13:12:45.626545 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:45.626547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:45.626691 (+ 144us) server_negotiation.cc:407] Connection header received
0928 13:12:45.626765 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:45.626765 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:45.626770 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:45.626820 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:45.627452 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:45.627453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:45.627454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:45.627457 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:45.627504 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:45.627508 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:45.627555 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:45.627644 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 13:12:45.627655 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 13:12:45.627792 30080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:45.627842 30080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3277072a21434d7e9021d06556de694c, State: Initialized, Role: FOLLOWER
I20260928 13:12:45.627961 30080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3277072a21434d7e9021d06556de694c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260928 13:12:45.628069 30080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:45.628106 30080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:45.628156 30080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:45.628798 30080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3277072a21434d7e9021d06556de694c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260928 13:12:45.628937 30080 leader_election.cc:302] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3277072a21434d7e9021d06556de694c; no voters:
I20260928 13:12:45.629109 30080 leader_election.cc:288] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:45.629187 30090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:45.629338 30090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [term 1 LEADER]: Becoming Leader. State: Replica: 3277072a21434d7e9021d06556de694c, State: Running, Role: LEADER
I20260928 13:12:45.629375 30080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:45.629447 30090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3277072a21434d7e9021d06556de694c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } }
I20260928 13:12:45.629880 30091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3277072a21434d7e9021d06556de694c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3277072a21434d7e9021d06556de694c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } } }
I20260928 13:12:45.629911 30092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3277072a21434d7e9021d06556de694c. Latest consensus state: current_term: 1 leader_uuid: "3277072a21434d7e9021d06556de694c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3277072a21434d7e9021d06556de694c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35297 } } }
I20260928 13:12:45.630105 30091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:45.630117 30092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c [sys.catalog]: This master's current role is: LEADER
W20260928 13:12:45.631145 30106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:12:45.631211 30106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:12:45.631263 30099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:45.631413 30099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:45.632823 30099 catalog_manager.cc:1347] Generated new cluster ID: 567898c968bf4e6299117e987167e1fe
I20260928 13:12:45.632890 30099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:45.648644 30099 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:45.648851 30099 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:45.652765 30099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3277072a21434d7e9021d06556de694c: Generated new TSK 0
I20260928 13:12:45.653007 30099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:45.659057 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 30012
2026-09-28T13:12:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:45 dist-test-slave-p1lc krb5kdc[30111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:45 dist-test-slave-p1lc krb5kdc[30111](info): set up 2 sockets
Sep 28 13:12:45 dist-test-slave-p1lc krb5kdc[30111](info): commencing operation
krb5kdc: starting...
W20260928 13:12:46.705155 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:46 dist-test-slave-p1lc krb5kdc[30111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:12:46.832515 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:33405
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:44947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:33405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:46.917850 30124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:46.918087 30124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:46.918111 30124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:46.919245 30124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:46.919281 30124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:46.919294 30124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:46.919307 30124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:46.919319 30124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:46.920722 30124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:44947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:33405
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:33405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:46.920995 30124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:46.921190 30124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:46.926218 30130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:46.926225 30129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:46.926348 30124 server_base.cc:1034] running on GCE node
W20260928 13:12:46.926225 30132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:46.926630 30124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:46.926879 30124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:46.928069 30124 hybrid_clock.cc:648] HybridClock initialized: now 1790601166928083 us; error 64 us; skew 500 ppm
Sep 28 13:12:46 dist-test-slave-p1lc krb5kdc[30111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601166, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:46.930869 30124 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:12:46.931428 30124 webserver.cc:460] Webserver started at http://127.8.88.190:34535/ using document root <none> and password file <none>
I20260928 13:12:46.931679 30124 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:46.931758 30124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:46.931913 30124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:46.933115 30124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "aff36ae849d8427d9f7f325ca5bfe011"
format_stamp: "Formatted at 2026-09-28 13:12:46 on dist-test-slave-p1lc"
server_key: "ecfbdae4818a2091f861d16b0ed7e332"
server_key_iv: "f083f659089b8fdd91b929ae52f2efc8"
server_key_version: "encryptionkey@0"
I20260928 13:12:46.933501 30124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "aff36ae849d8427d9f7f325ca5bfe011"
format_stamp: "Formatted at 2026-09-28 13:12:46 on dist-test-slave-p1lc"
server_key: "ecfbdae4818a2091f861d16b0ed7e332"
server_key_iv: "f083f659089b8fdd91b929ae52f2efc8"
server_key_version: "encryptionkey@0"
I20260928 13:12:46.934911 30124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:12:46.935967 30139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:46.936251 30124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:46.936355 30124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "aff36ae849d8427d9f7f325ca5bfe011"
format_stamp: "Formatted at 2026-09-28 13:12:46 on dist-test-slave-p1lc"
server_key: "ecfbdae4818a2091f861d16b0ed7e332"
server_key_iv: "f083f659089b8fdd91b929ae52f2efc8"
server_key_version: "encryptionkey@0"
I20260928 13:12:46.936444 30124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:46.957736 30124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:46.960090 30124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:46.960283 30124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:46.964481 30124 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:33405
I20260928 13:12:46.964545 30191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:33405 every 8 connection(s)
I20260928 13:12:46.964952 30124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:46.965900 30192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:46.967909 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 30124
I20260928 13:12:46.968082 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:46.968314 8546 external_mini_cluster.cc:1428] Setting key c6d1f0ceaba00abbd24bfb4124fdc918
I20260928 13:12:46.968488 30192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011: Bootstrap starting.
I20260928 13:12:46.969295 30192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:46.969606 30192 log.cc:824] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:46.970332 30192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011: No bootstrap required, opened a new log
I20260928 13:12:46.972239 30192 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff36ae849d8427d9f7f325ca5bfe011" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33405 } }
I20260928 13:12:46.972411 30192 raft_consensus.cc:374] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:46.972473 30192 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff36ae849d8427d9f7f325ca5bfe011, State: Initialized, Role: FOLLOWER
I20260928 13:12:46.972607 30192 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff36ae849d8427d9f7f325ca5bfe011" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33405 } }
I20260928 13:12:46.972710 30192 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:46.972756 30192 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:46.972805 30192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:46.973423 30192 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff36ae849d8427d9f7f325ca5bfe011" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33405 } }
I20260928 13:12:46.973548 30192 leader_election.cc:302] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff36ae849d8427d9f7f325ca5bfe011; no voters:
I20260928 13:12:46.973763 30192 leader_election.cc:288] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:46.973785 30197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:46.973963 30197 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [term 1 LEADER]: Becoming Leader. State: Replica: aff36ae849d8427d9f7f325ca5bfe011, State: Running, Role: LEADER
I20260928 13:12:46.974121 30192 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:46.974099 30197 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff36ae849d8427d9f7f325ca5bfe011" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33405 } }
I20260928 13:12:46.975631 30198 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aff36ae849d8427d9f7f325ca5bfe011" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff36ae849d8427d9f7f325ca5bfe011" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33405 } } }
Sep 28 13:12:46 dist-test-slave-p1lc krb5kdc[30111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:46.975811 30198 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:46.975978 30199 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff36ae849d8427d9f7f325ca5bfe011. Latest consensus state: current_term: 1 leader_uuid: "aff36ae849d8427d9f7f325ca5bfe011" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff36ae849d8427d9f7f325ca5bfe011" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 33405 } } }
I20260928 13:12:46.976668 30199 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011 [sys.catalog]: This master's current role is: LEADER
W20260928 13:12:46.976205 30212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:12:46.976791 30212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:12:46.976994 30214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:46.977303 30214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:46.979097 30214 catalog_manager.cc:1347] Generated new cluster ID: d749db002631401691aed380267dfced
I20260928 13:12:46.979161 30214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:46.981395 30195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:46.968974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51343 (local address 127.8.88.190:33405)
0928 13:12:46.969219 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:46.969227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:46.969256 (+ 29us) server_negotiation.cc:407] Connection header received
0928 13:12:46.969840 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:46.969851 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:46.969915 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:46.970099 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:46.970901 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:46.971897 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:46.972902 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:46.973374 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:46.973523 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:46.976907 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:46.976912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:46.976912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:46.976926 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:46.979162 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:46.979727 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:46.979730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:46.979731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:46.979841 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:46.980068 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:46.980069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:46.980069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:46.980378 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:46.980544 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:46.981041 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 13:12:46.981152 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
I20260928 13:12:46.982407 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:46.991770 30195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:46.984141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54731 (local address 127.8.88.190:33405)
0928 13:12:46.984179 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:46.984181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:46.984296 (+ 115us) server_negotiation.cc:407] Connection header received
0928 13:12:46.984381 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:46.984382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:46.984391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:46.984418 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:46.985199 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:46.985956 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:46.986957 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:46.987340 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:46.987378 (+ 38us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:46.988961 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:46.988962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:46.988963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:46.988970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:46.990535 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:46.990933 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:46.990934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:46.990934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:46.990985 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:46.991192 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:46.991193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:46.991194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:46.991311 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:46.991387 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:46.991602 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 13:12:46.991673 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:12:47.013654 30214 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:47.014264 30214 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:47.018728 30214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aff36ae849d8427d9f7f325ca5bfe011: Generated new TSK 0
I20260928 13:12:47.018946 30214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:47.025228 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 30124
W20260928 13:12:47.030103 30109 connection.cc:535] client connection to 127.8.88.190:33405 recv error: Network error: failed to read from TLS socket (remote: 127.8.88.190:33405): Connection reset by peer (error 104)
2026-09-28T13:12:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:47 dist-test-slave-p1lc krb5kdc[30223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:47 dist-test-slave-p1lc krb5kdc[30223](info): set up 2 sockets
Sep 28 13:12:47 dist-test-slave-p1lc krb5kdc[30223](info): commencing operation
krb5kdc: starting...
W20260928 13:12:48.069180 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:48 dist-test-slave-p1lc krb5kdc[30223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:12:48.200837 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:36305
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:36305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:48.285950 30236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:48.286186 30236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:48.286213 30236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:48.287420 30236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:48.287465 30236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:48.287479 30236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:48.287492 30236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:48.287505 30236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:48.289178 30236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:36305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:36305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:48.289449 30236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:48.289686 30236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:48.294745 30242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:48.294859 30236 server_base.cc:1034] running on GCE node
W20260928 13:12:48.294934 30244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:48.294778 30241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:48.295329 30236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:48.295677 30236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:48.296916 30236 hybrid_clock.cc:648] HybridClock initialized: now 1790601168296844 us; error 39 us; skew 500 ppm
Sep 28 13:12:48 dist-test-slave-p1lc krb5kdc[30223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601168, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:48.299860 30236 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:12:48.300416 30236 webserver.cc:460] Webserver started at http://127.8.88.190:39013/ using document root <none> and password file <none>
I20260928 13:12:48.300662 30236 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:48.300709 30236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:48.300810 30236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:48.302030 30236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "7f0122cdcc9f438285b30f29f32d6790"
format_stamp: "Formatted at 2026-09-28 13:12:48 on dist-test-slave-p1lc"
server_key: "95dfcdfc272f058a229403e4549ac291"
server_key_iv: "b29f4cbc654ff39e6c1d6e0f200c1dfa"
server_key_version: "encryptionkey@0"
I20260928 13:12:48.302378 30236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "7f0122cdcc9f438285b30f29f32d6790"
format_stamp: "Formatted at 2026-09-28 13:12:48 on dist-test-slave-p1lc"
server_key: "95dfcdfc272f058a229403e4549ac291"
server_key_iv: "b29f4cbc654ff39e6c1d6e0f200c1dfa"
server_key_version: "encryptionkey@0"
I20260928 13:12:48.303850 30236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:48.304795 30251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:48.305022 30236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:48.305146 30236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "7f0122cdcc9f438285b30f29f32d6790"
format_stamp: "Formatted at 2026-09-28 13:12:48 on dist-test-slave-p1lc"
server_key: "95dfcdfc272f058a229403e4549ac291"
server_key_iv: "b29f4cbc654ff39e6c1d6e0f200c1dfa"
server_key_version: "encryptionkey@0"
I20260928 13:12:48.305238 30236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:48.314237 30236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:48.316807 30236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:48.317051 30236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:48.321382 30236 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:36305
I20260928 13:12:48.321420 30303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:36305 every 8 connection(s)
I20260928 13:12:48.321805 30236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:48.322702 30304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:48.325075 30304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790: Bootstrap starting.
I20260928 13:12:48.325670 30304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:48.325984 30304 log.cc:824] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:48.326126 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 30236
I20260928 13:12:48.326249 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:48.326465 8546 external_mini_cluster.cc:1428] Setting key bff5e7d60d052fa008be29ce7eb0e8bb
I20260928 13:12:48.326818 30304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790: No bootstrap required, opened a new log
I20260928 13:12:48.329866 30304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f0122cdcc9f438285b30f29f32d6790" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36305 } }
I20260928 13:12:48.330169 30304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:48.330220 30304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f0122cdcc9f438285b30f29f32d6790, State: Initialized, Role: FOLLOWER
I20260928 13:12:48.330363 30304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f0122cdcc9f438285b30f29f32d6790" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36305 } }
I20260928 13:12:48.330461 30304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:48.330500 30304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:48.330554 30304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:48.331177 30304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f0122cdcc9f438285b30f29f32d6790" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36305 } }
I20260928 13:12:48.331266 30304 leader_election.cc:302] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f0122cdcc9f438285b30f29f32d6790; no voters:
I20260928 13:12:48.331537 30304 leader_election.cc:288] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:48.331555 30309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:48.331696 30309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [term 1 LEADER]: Becoming Leader. State: Replica: 7f0122cdcc9f438285b30f29f32d6790, State: Running, Role: LEADER
I20260928 13:12:48.331828 30304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:48.331976 30309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f0122cdcc9f438285b30f29f32d6790" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36305 } }
I20260928 13:12:48.332435 30310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f0122cdcc9f438285b30f29f32d6790. Latest consensus state: current_term: 1 leader_uuid: "7f0122cdcc9f438285b30f29f32d6790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f0122cdcc9f438285b30f29f32d6790" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36305 } } }
I20260928 13:12:48.332597 30310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:48.333118 30320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:48.333472 30311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f0122cdcc9f438285b30f29f32d6790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f0122cdcc9f438285b30f29f32d6790" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 36305 } } }
I20260928 13:12:48.333626 30320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:48.333729 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:48.327000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51673 (local address 127.8.88.190:36305)
0928 13:12:48.327175 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:48.327186 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:48.327221 (+ 35us) server_negotiation.cc:407] Connection header received
0928 13:12:48.328292 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:48.328302 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:48.328368 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:48.328535 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:48.328788 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:48.330387 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:48.331348 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:48.331965 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:48.332084 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:48.332919 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:48.332923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:48.332923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:48.333144 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:48.333184 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:48.333199 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:48.333284 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:48.333448 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 13:12:48.333488 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260928 13:12:48.333554 30311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:48.334686 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:48.336374 30320 catalog_manager.cc:1347] Generated new cluster ID: edda5a115c0e4fae82f7d4faa3038daf
I20260928 13:12:48.336432 30320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:48.339371 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:48.336226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42613 (local address 127.8.88.190:36305)
0928 13:12:48.336266 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:48.336269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:48.336484 (+ 215us) server_negotiation.cc:407] Connection header received
0928 13:12:48.336565 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:48.336566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:48.336576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:48.336632 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:48.336790 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:48.337322 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:48.338068 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:48.338291 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:48.338364 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:48.339114 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:48.339115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:48.339115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:48.339120 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:48.339139 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:48.339143 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:48.339199 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:48.339285 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 13:12:48.339296 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:12:48.360422 30320 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:48.361083 30320 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:48.366907 30320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f0122cdcc9f438285b30f29f32d6790: Generated new TSK 0
I20260928 13:12:48.367148 30320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:48.374735 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 30236
W20260928 13:12:48.379101 30221 connection.cc:535] client connection to 127.8.88.190:36305 recv error: Network error: failed to read from TLS socket (remote: 127.8.88.190:36305): Connection reset by peer (error 104)
2026-09-28T13:12:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:48 dist-test-slave-p1lc krb5kdc[30335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:48 dist-test-slave-p1lc krb5kdc[30335](info): set up 2 sockets
Sep 28 13:12:48 dist-test-slave-p1lc krb5kdc[30335](info): commencing operation
krb5kdc: starting...
W20260928 13:12:49.437163 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:49 dist-test-slave-p1lc krb5kdc[30335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:12:49.567636 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:39891
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:37383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:39891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:49.649768 30348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:49.650050 30348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:49.650092 30348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:49.651312 30348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:49.651396 30348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:49.651420 30348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:49.651448 30348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:49.651474 30348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:49.653002 30348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:37383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:39891
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:39891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:49.653302 30348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:49.653535 30348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:49.658473 30353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:49.658490 30354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:49.658664 30348 server_base.cc:1034] running on GCE node
W20260928 13:12:49.658502 30356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:49.658969 30348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:49.659276 30348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:49.660426 30348 hybrid_clock.cc:648] HybridClock initialized: now 1790601169660407 us; error 40 us; skew 500 ppm
Sep 28 13:12:49 dist-test-slave-p1lc krb5kdc[30335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601169, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:49.679915 30348 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:12:49.680459 30348 webserver.cc:460] Webserver started at http://127.8.88.190:42545/ using document root <none> and password file <none>
I20260928 13:12:49.680722 30348 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:49.680794 30348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:49.680969 30348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:49.682106 30348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "c2c444f3daa94886b2214ea5f7555e3f"
format_stamp: "Formatted at 2026-09-28 13:12:49 on dist-test-slave-p1lc"
server_key: "1e3a74e93144160edfc1b48728134462"
server_key_iv: "5bf9a33214c79ff454126d2861ec18d9"
server_key_version: "encryptionkey@0"
I20260928 13:12:49.682572 30348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "c2c444f3daa94886b2214ea5f7555e3f"
format_stamp: "Formatted at 2026-09-28 13:12:49 on dist-test-slave-p1lc"
server_key: "1e3a74e93144160edfc1b48728134462"
server_key_iv: "5bf9a33214c79ff454126d2861ec18d9"
server_key_version: "encryptionkey@0"
I20260928 13:12:49.684342 30348 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 13:12:49.685336 30363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:49.685578 30348 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:49.685694 30348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "c2c444f3daa94886b2214ea5f7555e3f"
format_stamp: "Formatted at 2026-09-28 13:12:49 on dist-test-slave-p1lc"
server_key: "1e3a74e93144160edfc1b48728134462"
server_key_iv: "5bf9a33214c79ff454126d2861ec18d9"
server_key_version: "encryptionkey@0"
I20260928 13:12:49.685774 30348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:49.689355 30348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:49.691668 30348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:49.691893 30348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:49.696600 30348 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:39891
I20260928 13:12:49.696631 30415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:39891 every 8 connection(s)
I20260928 13:12:49.697034 30348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:49.698118 30416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:49.700325 30416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f: Bootstrap starting.
I20260928 13:12:49.700929 30416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:49.701208 30416 log.cc:824] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:49.701920 30416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f: No bootstrap required, opened a new log
I20260928 13:12:49.703272 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 30348
I20260928 13:12:49.703397 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:49.703598 8546 external_mini_cluster.cc:1428] Setting key 34105ec31b6e3c24f5eb9ead02396e48
I20260928 13:12:49.703888 30416 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c444f3daa94886b2214ea5f7555e3f" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39891 } }
I20260928 13:12:49.704087 30416 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:49.704154 30416 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2c444f3daa94886b2214ea5f7555e3f, State: Initialized, Role: FOLLOWER
I20260928 13:12:49.704295 30416 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c444f3daa94886b2214ea5f7555e3f" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39891 } }
I20260928 13:12:49.704412 30416 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:49.704461 30416 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:49.704509 30416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:49.705346 30416 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c444f3daa94886b2214ea5f7555e3f" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39891 } }
I20260928 13:12:49.705457 30416 leader_election.cc:302] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2c444f3daa94886b2214ea5f7555e3f; no voters:
I20260928 13:12:49.705651 30416 leader_election.cc:288] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:49.705856 30422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:49.705929 30416 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:49.706092 30422 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [term 1 LEADER]: Becoming Leader. State: Replica: c2c444f3daa94886b2214ea5f7555e3f, State: Running, Role: LEADER
I20260928 13:12:49.706246 30420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:49.704200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36447 (local address 127.8.88.190:39891)
0928 13:12:49.704411 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:49.704421 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:49.704459 (+ 38us) server_negotiation.cc:407] Connection header received
0928 13:12:49.704603 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:49.704611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:49.704643 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:49.704801 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:49.705709 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:49.705714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:49.705714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:49.705719 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:49.705769 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:49.705786 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:49.705857 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:49.705983 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 13:12:49.706033 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":116,"threads_started":1}
I20260928 13:12:49.706229 30422 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c444f3daa94886b2214ea5f7555e3f" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39891 } }
I20260928 13:12:49.707065 30421 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2c444f3daa94886b2214ea5f7555e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c444f3daa94886b2214ea5f7555e3f" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39891 } } }
I20260928 13:12:49.707152 30421 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:49.707367 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:49.707360 30423 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2c444f3daa94886b2214ea5f7555e3f. Latest consensus state: current_term: 1 leader_uuid: "c2c444f3daa94886b2214ea5f7555e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2c444f3daa94886b2214ea5f7555e3f" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 39891 } } }
I20260928 13:12:49.707696 30423 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f [sys.catalog]: This master's current role is: LEADER
W20260928 13:12:49.708063 30436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:12:49.708125 30436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:12:49.708214 30435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:49.708446 30435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:49.710265 30435 catalog_manager.cc:1347] Generated new cluster ID: 51005542e01f4003b5e68146ea6987e4
I20260928 13:12:49.710328 30435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:49.710327 30420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:49.709012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49273 (local address 127.8.88.190:39891)
0928 13:12:49.709052 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:49.709054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:49.709171 (+ 117us) server_negotiation.cc:407] Connection header received
0928 13:12:49.709230 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:49.709231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:49.709249 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:49.709311 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:49.709947 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:49.709948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:49.709949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:49.709953 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:49.710058 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:49.710063 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:49.710135 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:49.710198 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 13:12:49.710214 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:12:49.728060 30435 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:49.728228 30435 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:49.737867 30435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2c444f3daa94886b2214ea5f7555e3f: Generated new TSK 0
I20260928 13:12:49.738106 30435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:49.740773 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 30348
2026-09-28T13:12:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1366 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:49 dist-test-slave-p1lc krb5kdc[30447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:49 dist-test-slave-p1lc krb5kdc[30447](info): set up 2 sockets
Sep 28 13:12:49 dist-test-slave-p1lc krb5kdc[30447](info): commencing operation
krb5kdc: starting...
W20260928 13:12:50.785413 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:50 dist-test-slave-p1lc krb5kdc[30447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:12:50.915521 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:40705
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:45487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:40705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:51.001565 30460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:51.001806 30460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:51.001832 30460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:51.003069 30460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:51.003132 30460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:51.003149 30460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:51.003163 30460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:51.003176 30460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:51.004724 30460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:45487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:40705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:40705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:51.005028 30460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:51.005249 30460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:51.010669 30468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:51.010655 30465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:51.010655 30466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:51.010876 30460 server_base.cc:1034] running on GCE node
I20260928 13:12:51.011234 30460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:51.011561 30460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:51.012740 30460 hybrid_clock.cc:648] HybridClock initialized: now 1790601171012691 us; error 73 us; skew 500 ppm
Sep 28 13:12:51 dist-test-slave-p1lc krb5kdc[30447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601171, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:51.015538 30460 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:12:51.016263 30460 webserver.cc:460] Webserver started at http://127.8.88.190:36883/ using document root <none> and password file <none>
I20260928 13:12:51.016549 30460 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:51.016624 30460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:51.016778 30460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:51.017967 30460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "5acc44f3b26046a1a7f12b4d08ee708a"
format_stamp: "Formatted at 2026-09-28 13:12:51 on dist-test-slave-p1lc"
server_key: "fc04ed9d5285a1efd13cb29e017fa8a1"
server_key_iv: "0f60b0349d7e38b73e55c8463212e308"
server_key_version: "encryptionkey@0"
I20260928 13:12:51.018363 30460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "5acc44f3b26046a1a7f12b4d08ee708a"
format_stamp: "Formatted at 2026-09-28 13:12:51 on dist-test-slave-p1lc"
server_key: "fc04ed9d5285a1efd13cb29e017fa8a1"
server_key_iv: "0f60b0349d7e38b73e55c8463212e308"
server_key_version: "encryptionkey@0"
I20260928 13:12:51.019788 30460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:51.020677 30475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:51.021013 30460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:51.021174 30460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "5acc44f3b26046a1a7f12b4d08ee708a"
format_stamp: "Formatted at 2026-09-28 13:12:51 on dist-test-slave-p1lc"
server_key: "fc04ed9d5285a1efd13cb29e017fa8a1"
server_key_iv: "0f60b0349d7e38b73e55c8463212e308"
server_key_version: "encryptionkey@0"
I20260928 13:12:51.021270 30460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:51.043287 30460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:51.045883 30460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:51.046011 30460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:51.050151 30460 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:40705
I20260928 13:12:51.050190 30527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:40705 every 8 connection(s)
I20260928 13:12:51.050552 30460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:51.051216 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 30460
I20260928 13:12:51.051343 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:51.051555 8546 external_mini_cluster.cc:1428] Setting key d62ec7b778af8bc5fb1698b42b55828b
I20260928 13:12:51.051695 30528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:51.055546 30528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a: Bootstrap starting.
I20260928 13:12:51.056237 30528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:51.056582 30528 log.cc:824] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:51.057463 30528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a: No bootstrap required, opened a new log
Sep 28 13:12:51 dist-test-slave-p1lc krb5kdc[30447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:51.059442 30528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5acc44f3b26046a1a7f12b4d08ee708a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 40705 } }
I20260928 13:12:51.059643 30528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:51.059690 30528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5acc44f3b26046a1a7f12b4d08ee708a, State: Initialized, Role: FOLLOWER
I20260928 13:12:51.060086 30528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5acc44f3b26046a1a7f12b4d08ee708a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 40705 } }
I20260928 13:12:51.060199 30528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:51.060392 30528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:51.060452 30528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:51.061343 30528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5acc44f3b26046a1a7f12b4d08ee708a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 40705 } }
I20260928 13:12:51.061465 30528 leader_election.cc:302] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5acc44f3b26046a1a7f12b4d08ee708a; no voters:
I20260928 13:12:51.061687 30528 leader_election.cc:288] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:51.061722 30533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:51.061831 30533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [term 1 LEADER]: Becoming Leader. State: Replica: 5acc44f3b26046a1a7f12b4d08ee708a, State: Running, Role: LEADER
I20260928 13:12:51.061995 30528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:51.061982 30533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5acc44f3b26046a1a7f12b4d08ee708a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 40705 } }
I20260928 13:12:51.062537 30535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5acc44f3b26046a1a7f12b4d08ee708a. Latest consensus state: current_term: 1 leader_uuid: "5acc44f3b26046a1a7f12b4d08ee708a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5acc44f3b26046a1a7f12b4d08ee708a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 40705 } } }
I20260928 13:12:51.062626 30535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:51.062821 30534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5acc44f3b26046a1a7f12b4d08ee708a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5acc44f3b26046a1a7f12b4d08ee708a" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 40705 } } }
I20260928 13:12:51.063373 30538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:51.063493 30534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:51.064051 30538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:51.064728 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:51.052314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43091 (local address 127.8.88.190:40705)
0928 13:12:51.052522 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:51.052533 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:51.052567 (+ 34us) server_negotiation.cc:407] Connection header received
0928 13:12:51.053164 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:51.053173 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:51.053240 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:51.053408 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:51.054059 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:51.055453 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:51.056575 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:51.057236 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:51.057351 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:51.060165 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:51.060170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:51.060170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:51.060183 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:51.062515 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:51.063064 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:51.063067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:51.063068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:51.063173 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:51.063432 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:51.063433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:51.063434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:51.063795 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:51.063961 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:51.064416 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 13:12:51.064548 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
I20260928 13:12:51.065721 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:51.065723 30538 catalog_manager.cc:1347] Generated new cluster ID: 965e43501a294898a783e19c417eef87
I20260928 13:12:51.065773 30538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:51.075354 30538 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:51.076084 30538 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:51.076622 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:51.069470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53309 (local address 127.8.88.190:40705)
0928 13:12:51.069513 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:51.069515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:51.069716 (+ 201us) server_negotiation.cc:407] Connection header received
0928 13:12:51.069794 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:51.069795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:51.069806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:51.069870 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:51.070437 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:51.070963 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:51.071703 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:51.071933 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:51.072010 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:51.073666 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:51.073668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:51.073668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:51.073709 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:51.075408 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:51.075793 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:51.075794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:51.075795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:51.075853 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:51.076062 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:51.076063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:51.076064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:51.076215 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:51.076311 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:51.076427 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 13:12:51.076518 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:12:51.087486 30538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5acc44f3b26046a1a7f12b4d08ee708a: Generated new TSK 0
I20260928 13:12:51.087713 30538 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:51.089783 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 30460
2026-09-28T13:12:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:51 dist-test-slave-p1lc krb5kdc[30559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:51 dist-test-slave-p1lc krb5kdc[30559](info): set up 2 sockets
Sep 28 13:12:51 dist-test-slave-p1lc krb5kdc[30559](info): commencing operation
krb5kdc: starting...
W20260928 13:12:52.177203 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:52 dist-test-slave-p1lc krb5kdc[30559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:12:52.305369 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:42357
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:40703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:42357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:52.389039 30572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:52.389273 30572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:52.389297 30572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:52.390612 30572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:52.390659 30572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:52.390674 30572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:52.390687 30572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:52.390699 30572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:52.392326 30572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:40703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:42357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:42357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:52.392606 30572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:52.392841 30572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:52.397929 30577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:52.397938 30578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:52.397938 30580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:52.398288 30572 server_base.cc:1034] running on GCE node
I20260928 13:12:52.398566 30572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:52.398873 30572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:52.400027 30572 hybrid_clock.cc:648] HybridClock initialized: now 1790601172400000 us; error 52 us; skew 500 ppm
Sep 28 13:12:52 dist-test-slave-p1lc krb5kdc[30559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601172, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:52.402812 30572 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:12:52.403375 30572 webserver.cc:460] Webserver started at http://127.8.88.190:40895/ using document root <none> and password file <none>
I20260928 13:12:52.403611 30572 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:52.403681 30572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:52.403836 30572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:52.404989 30572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "1110e688463f427689fdcb8b45951c4b"
format_stamp: "Formatted at 2026-09-28 13:12:52 on dist-test-slave-p1lc"
server_key: "1495a3c7614486ed74502280dcc9e1b0"
server_key_iv: "be01c34490b70c3c76a62a0fb4cbd3df"
server_key_version: "encryptionkey@0"
I20260928 13:12:52.405361 30572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "1110e688463f427689fdcb8b45951c4b"
format_stamp: "Formatted at 2026-09-28 13:12:52 on dist-test-slave-p1lc"
server_key: "1495a3c7614486ed74502280dcc9e1b0"
server_key_iv: "be01c34490b70c3c76a62a0fb4cbd3df"
server_key_version: "encryptionkey@0"
I20260928 13:12:52.406726 30572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:12:52.407567 30587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:52.407763 30572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:52.407863 30572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "1110e688463f427689fdcb8b45951c4b"
format_stamp: "Formatted at 2026-09-28 13:12:52 on dist-test-slave-p1lc"
server_key: "1495a3c7614486ed74502280dcc9e1b0"
server_key_iv: "be01c34490b70c3c76a62a0fb4cbd3df"
server_key_version: "encryptionkey@0"
I20260928 13:12:52.407948 30572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:52.419767 30572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:52.422096 30572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:52.422273 30572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:52.426465 30572 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:42357
I20260928 13:12:52.426535 30639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:42357 every 8 connection(s)
I20260928 13:12:52.426909 30572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:52.427870 30640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:52.430239 30640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b: Bootstrap starting.
I20260928 13:12:52.430787 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 30572
I20260928 13:12:52.430905 30640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:52.430912 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:52.431114 8546 external_mini_cluster.cc:1428] Setting key 3ebf89ed4b6eacc75e7a08aaf6e3cb9a
I20260928 13:12:52.431241 30640 log.cc:824] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:52.432085 30640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b: No bootstrap required, opened a new log
I20260928 13:12:52.434198 30640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1110e688463f427689fdcb8b45951c4b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 42357 } }
I20260928 13:12:52.434367 30640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:52.434405 30640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1110e688463f427689fdcb8b45951c4b, State: Initialized, Role: FOLLOWER
I20260928 13:12:52.434595 30640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1110e688463f427689fdcb8b45951c4b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 42357 } }
I20260928 13:12:52.434695 30640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:52.434741 30640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:52.434790 30640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:52.435395 30640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1110e688463f427689fdcb8b45951c4b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 42357 } }
I20260928 13:12:52.435519 30640 leader_election.cc:302] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1110e688463f427689fdcb8b45951c4b; no voters:
I20260928 13:12:52.435719 30640 leader_election.cc:288] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:52.435768 30645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:52.435914 30645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [term 1 LEADER]: Becoming Leader. State: Replica: 1110e688463f427689fdcb8b45951c4b, State: Running, Role: LEADER
I20260928 13:12:52.435999 30640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:52.436012 30645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1110e688463f427689fdcb8b45951c4b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 42357 } }
I20260928 13:12:52.436389 30646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1110e688463f427689fdcb8b45951c4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1110e688463f427689fdcb8b45951c4b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 42357 } } }
I20260928 13:12:52.436389 30647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1110e688463f427689fdcb8b45951c4b. Latest consensus state: current_term: 1 leader_uuid: "1110e688463f427689fdcb8b45951c4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1110e688463f427689fdcb8b45951c4b" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 42357 } } }
I20260928 13:12:52.436498 30646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:52.436502 30647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:52.436796 30649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:52.437134 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:52.431673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46435 (local address 127.8.88.190:42357)
0928 13:12:52.431849 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:52.431859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:52.431894 (+ 35us) server_negotiation.cc:407] Connection header received
0928 13:12:52.432407 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:52.432414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:52.432463 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:52.432609 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:52.432811 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:52.433792 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:52.434689 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:52.435209 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:52.435393 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:52.436217 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:52.436221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:52.436221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:52.436446 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:52.436484 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:52.436501 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:52.436575 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:52.436834 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 13:12:52.436935 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260928 13:12:52.437434 30649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:52.438163 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:52.439234 30649 catalog_manager.cc:1347] Generated new cluster ID: b9f11ea58026458b86eb38d260bb6c78
I20260928 13:12:52.439296 30649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:52.443254 30644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:52.439771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36761 (local address 127.8.88.190:42357)
0928 13:12:52.439808 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:52.439810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:52.439905 (+ 95us) server_negotiation.cc:407] Connection header received
0928 13:12:52.439965 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:52.439966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:52.439975 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:52.440028 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:52.440124 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:52.440831 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:52.441858 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:52.442167 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:52.442229 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:52.442990 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:52.442992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:52.442992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:52.442996 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:52.443011 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:52.443015 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:52.443075 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:52.443154 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 13:12:52.443176 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:12:52.444973 30649 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:52.445508 30649 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:52.448786 30649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1110e688463f427689fdcb8b45951c4b: Generated new TSK 0
I20260928 13:12:52.449003 30649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:52.457317 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 30572
2026-09-28T13:12:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:52 dist-test-slave-p1lc krb5kdc[30671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:52 dist-test-slave-p1lc krb5kdc[30671](info): set up 2 sockets
Sep 28 13:12:52 dist-test-slave-p1lc krb5kdc[30671](info): commencing operation
krb5kdc: starting...
W20260928 13:12:53.513240 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:53 dist-test-slave-p1lc krb5kdc[30671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:12:53.640726 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:34753
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:39211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:34753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:53.726583 30684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:53.726805 30684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:53.726830 30684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:53.728046 30684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:53.728096 30684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:53.728109 30684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:53.728122 30684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:53.728134 30684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:53.729725 30684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:39211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:34753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:34753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:53.730000 30684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:53.730250 30684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:53.735404 30690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:53.735482 30692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:53.735508 30689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:53.735499 30684 server_base.cc:1034] running on GCE node
I20260928 13:12:53.735991 30684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:53.736299 30684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:53.737449 30684 hybrid_clock.cc:648] HybridClock initialized: now 1790601173737428 us; error 42 us; skew 500 ppm
Sep 28 13:12:53 dist-test-slave-p1lc krb5kdc[30671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601173, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:53.740401 30684 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:12:53.741015 30684 webserver.cc:460] Webserver started at http://127.8.88.190:46437/ using document root <none> and password file <none>
I20260928 13:12:53.741277 30684 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:53.741353 30684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:53.741518 30684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:53.742723 30684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "f9dc64ca409048ddb98f81c5514e8c13"
format_stamp: "Formatted at 2026-09-28 13:12:53 on dist-test-slave-p1lc"
server_key: "d2cbcb7510c58010bfe236da795604eb"
server_key_iv: "3b8a8dbae41d1b8f83df69bf25382244"
server_key_version: "encryptionkey@0"
I20260928 13:12:53.743120 30684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "f9dc64ca409048ddb98f81c5514e8c13"
format_stamp: "Formatted at 2026-09-28 13:12:53 on dist-test-slave-p1lc"
server_key: "d2cbcb7510c58010bfe236da795604eb"
server_key_iv: "3b8a8dbae41d1b8f83df69bf25382244"
server_key_version: "encryptionkey@0"
I20260928 13:12:53.744666 30684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:53.745626 30699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:53.745868 30684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:53.745988 30684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "f9dc64ca409048ddb98f81c5514e8c13"
format_stamp: "Formatted at 2026-09-28 13:12:53 on dist-test-slave-p1lc"
server_key: "d2cbcb7510c58010bfe236da795604eb"
server_key_iv: "3b8a8dbae41d1b8f83df69bf25382244"
server_key_version: "encryptionkey@0"
I20260928 13:12:53.746074 30684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:53.749934 30684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:53.752560 30684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:53.752744 30684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:53.757321 30684 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:34753
I20260928 13:12:53.757356 30751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:34753 every 8 connection(s)
I20260928 13:12:53.757726 30684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:53.758822 30752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:53.761270 30752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13: Bootstrap starting.
I20260928 13:12:53.761863 30752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:53.762168 30752 log.cc:824] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:53.762946 30752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13: No bootstrap required, opened a new log
I20260928 13:12:53.764545 30752 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9dc64ca409048ddb98f81c5514e8c13" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 34753 } }
I20260928 13:12:53.764706 30752 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 13:12:53.764730 30752 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9dc64ca409048ddb98f81c5514e8c13, State: Initialized, Role: FOLLOWER
I20260928 13:12:53.764935 30752 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9dc64ca409048ddb98f81c5514e8c13" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 34753 } }
I20260928 13:12:53.765010 30752 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 13:12:53.765033 30752 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 13:12:53.765102 30752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:53.766017 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 30684
I20260928 13:12:53.766062 30752 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9dc64ca409048ddb98f81c5514e8c13" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 34753 } }
I20260928 13:12:53.766173 30752 leader_election.cc:302] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9dc64ca409048ddb98f81c5514e8c13; no voters:
I20260928 13:12:53.766201 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:53.766371 30752 leader_election.cc:288] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 13:12:53.766397 30755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:53.766497 8546 external_mini_cluster.cc:1428] Setting key f8e1e15f3aefaa3a95c81cf0537c2ec1
I20260928 13:12:53.766510 30755 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [term 1 LEADER]: Becoming Leader. State: Replica: f9dc64ca409048ddb98f81c5514e8c13, State: Running, Role: LEADER
I20260928 13:12:53.766588 30755 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9dc64ca409048ddb98f81c5514e8c13" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 34753 } }
I20260928 13:12:53.766654 30752 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:53.766901 30756 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9dc64ca409048ddb98f81c5514e8c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9dc64ca409048ddb98f81c5514e8c13" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 34753 } } }
I20260928 13:12:53.767019 30756 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:53.767138 30757 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9dc64ca409048ddb98f81c5514e8c13. Latest consensus state: current_term: 1 leader_uuid: "f9dc64ca409048ddb98f81c5514e8c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9dc64ca409048ddb98f81c5514e8c13" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 34753 } } }
I20260928 13:12:53.767406 30757 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:53.767606 30762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:53.768430 30762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:53.769559 30764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:53.767242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33935 (local address 127.8.88.190:34753)
0928 13:12:53.767448 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:53.767461 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:53.767506 (+ 45us) server_negotiation.cc:407] Connection header received
0928 13:12:53.767634 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:53.767641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:53.767678 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:53.768275 (+ 597us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:53.769074 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:53.769078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:53.769079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:53.769084 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:53.769128 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:53.769155 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:53.769236 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:53.769319 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 13:12:53.769364 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20260928 13:12:53.770599 30762 catalog_manager.cc:1347] Generated new cluster ID: 932df8e789b94c32bc8413fb7993987f
I20260928 13:12:53.770650 30762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:53.774691 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:53.777539 30764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:53.776228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38831 (local address 127.8.88.190:34753)
0928 13:12:53.776271 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:53.776273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:53.776396 (+ 123us) server_negotiation.cc:407] Connection header received
0928 13:12:53.776474 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:53.776475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:53.776482 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:53.776527 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:53.777294 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:53.777295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:53.777296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 13:12:53.777300 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:53.777316 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 13:12:53.777320 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:53.777371 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:53.777424 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 13:12:53.777439 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:12:53.786530 30762 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:53.786753 30762 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:53.791622 30762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9dc64ca409048ddb98f81c5514e8c13: Generated new TSK 0
I20260928 13:12:53.791834 30762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:53.799330 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 30684
2026-09-28T13:12:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1341 ms)
[----------] 18 tests from AuthTokenIssuingTest (24284 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:53 dist-test-slave-p1lc krb5kdc[30783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:53 dist-test-slave-p1lc krb5kdc[30783](info): set up 2 sockets
Sep 28 13:12:53 dist-test-slave-p1lc krb5kdc[30783](info): commencing operation
krb5kdc: starting...
W20260928 13:12:54.849022 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:54 dist-test-slave-p1lc krb5kdc[30783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:12:54.981499 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:38155
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:38155,127.8.88.189:37051,127.8.88.188:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:55.065413 30796 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 13:12:55.065632 30796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:55.065650 30796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:55.065670 30796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:55.066932 30796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:55.066977 30796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:55.066992 30796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:55.067005 30796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:55.067070 30796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:55.068575 30796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:38155,127.8.88.189:37051,127.8.88.188:44179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:38155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:55.068835 30796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:55.069111 30796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:55.074261 30801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:55.074265 30802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:55.074280 30796 server_base.cc:1034] running on GCE node
W20260928 13:12:55.074261 30804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:55.074728 30796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:55.075024 30796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:55.076153 30796 hybrid_clock.cc:648] HybridClock initialized: now 1790601175076140 us; error 34 us; skew 500 ppm
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601175, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:55.087880 30796 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:12:55.088505 30796 webserver.cc:460] Webserver started at http://127.8.88.190:41509/ using document root <none> and password file <none>
I20260928 13:12:55.088773 30796 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:55.088821 30796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:55.089032 30796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:55.090142 30796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "3b6eceea186d41ae905b4658d52ee66c"
format_stamp: "Formatted at 2026-09-28 13:12:55 on dist-test-slave-p1lc"
server_key: "1f96a50978bd93c7eebfdbebe360aa86"
server_key_iv: "b27d59a2a14d2e8257ab8d47c41b816f"
server_key_version: "encryptionkey@0"
I20260928 13:12:55.090505 30796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "3b6eceea186d41ae905b4658d52ee66c"
format_stamp: "Formatted at 2026-09-28 13:12:55 on dist-test-slave-p1lc"
server_key: "1f96a50978bd93c7eebfdbebe360aa86"
server_key_iv: "b27d59a2a14d2e8257ab8d47c41b816f"
server_key_version: "encryptionkey@0"
I20260928 13:12:55.091850 30796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 13:12:55.092736 30811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:55.092944 30796 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:12:55.093041 30796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "3b6eceea186d41ae905b4658d52ee66c"
format_stamp: "Formatted at 2026-09-28 13:12:55 on dist-test-slave-p1lc"
server_key: "1f96a50978bd93c7eebfdbebe360aa86"
server_key_iv: "b27d59a2a14d2e8257ab8d47c41b816f"
server_key_version: "encryptionkey@0"
I20260928 13:12:55.093098 30796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:55.117759 30796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:55.120141 30796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:55.120261 30796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:55.124444 30796 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:38155
I20260928 13:12:55.124502 30863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:38155 every 8 connection(s)
I20260928 13:12:55.124845 30796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:55.125865 30864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:55.126873 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 30796
I20260928 13:12:55.126959 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
I20260928 13:12:55.127153 8546 external_mini_cluster.cc:1428] Setting key 35bc8f235297b9edc495f1c1c94a80ac
I20260928 13:12:55.127704 30864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601175, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.88.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.189@KRBTEST.COM" created.
I20260928 13:12:55.138928 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.128488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:38155 (local address 127.0.0.1:55390)
0928 13:12:55.128681 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.128726 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:55.129270 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:55.129605 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:55.129607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:55.129861 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:55.130017 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.130021 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.131620 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.131621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.132325 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.132327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.133243 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.133244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.133305 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.133906 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:55.133915 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:55.135387 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:55.137279 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.137286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.137288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.137488 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.137822 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.137823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.137824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.137923 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.138378 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:55.138385 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:55.138447 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 13:12:55.138532 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 13:12:55.138670 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260928 13:12:55.138967 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.128546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.8.88.190:38155)
0928 13:12:55.128726 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.128736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.128820 (+ 84us) server_negotiation.cc:407] Connection header received
0928 13:12:55.129447 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.129450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.129473 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.129620 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.130117 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.131517 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.132420 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.133040 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.133182 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.135475 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.135482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.135482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.135489 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.137188 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.137604 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.137607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.137607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.137693 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.138010 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.138011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.138011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.138290 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.138386 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.138612 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.138766 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
I20260928 13:12:55.139768 30864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:12:55.140137 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.140015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37051 (local address 127.0.0.1:53894)
0928 13:12:55.140066 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.140111 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 13:12:55.140292 30815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.88.189:37051: connect: Connection refused (error 111)
I20260928 13:12:55.140779 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.140685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37051 (local address 127.0.0.1:53910)
0928 13:12:55.140711 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.140734 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 13:12:55.140921 30864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.189:37051: Network error: Client connection negotiation failed: client connection to 127.8.88.189:37051: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.189.keytab.
Entry for principal kudu/127.8.88.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.189.keytab.
I20260928 13:12:55.157163 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.189:37051
--webserver_interface=127.8.88.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.189.keytab
--principal=kudu/127.8.88.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:38155,127.8.88.189:37051,127.8.88.188:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:12:55.190198 30864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } attempt: 1
I20260928 13:12:55.190759 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.190599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37051 (local address 127.0.0.1:53914)
0928 13:12:55.190692 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.190726 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 13:12:55.191179 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.191008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37051 (local address 127.0.0.1:53920)
0928 13:12:55.191124 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.191151 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260928 13:12:55.191267 30864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.189:37051: Network error: Client connection negotiation failed: client connection to 127.8.88.189:37051: connect: Connection refused (error 111)
W20260928 13:12:55.241376 30871 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 13:12:55.241634 30871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:55.241653 30871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:55.241676 30871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:55.242920 30871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:55.242971 30871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:55.242986 30871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:55.243000 30871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:55.243011 30871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:55.244572 30871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:38155,127.8.88.189:37051,127.8.88.188:44179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.189.keytab
--principal=kudu/127.8.88.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.189:37051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.88.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:55.244844 30871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:55.245115 30871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:55.250303 30877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:55.250332 30879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:55.250303 30876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:55.250872 30871 server_base.cc:1034] running on GCE node
I20260928 13:12:55.251150 30871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:55.251452 30871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:55.252702 30871 hybrid_clock.cc:648] HybridClock initialized: now 1790601175252680 us; error 41 us; skew 500 ppm
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601175, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:55.255404 30871 init.cc:377] Logged in from keytab as kudu/127.8.88.189@KRBTEST.COM (short username kudu)
I20260928 13:12:55.255911 30871 webserver.cc:460] Webserver started at http://127.8.88.189:34603/ using document root <none> and password file <none>
I20260928 13:12:55.256165 30871 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:55.256248 30871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:55.256402 30871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:55.257611 30871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/data/instance:
uuid: "d16f9adf05ca4b58861d3fffd8daa07c"
format_stamp: "Formatted at 2026-09-28 13:12:55 on dist-test-slave-p1lc"
server_key: "a33c074df6e6b40354710b850a50d237"
server_key_iv: "c091a4b5e537ac9ab4a2092c3a77b48c"
server_key_version: "encryptionkey@0"
I20260928 13:12:55.257992 30871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/wal/instance:
uuid: "d16f9adf05ca4b58861d3fffd8daa07c"
format_stamp: "Formatted at 2026-09-28 13:12:55 on dist-test-slave-p1lc"
server_key: "a33c074df6e6b40354710b850a50d237"
server_key_iv: "c091a4b5e537ac9ab4a2092c3a77b48c"
server_key_version: "encryptionkey@0"
I20260928 13:12:55.259392 30871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:12:55.259403 30864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } attempt: 2
I20260928 13:12:55.259819 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.259670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37051 (local address 127.0.0.1:53930)
0928 13:12:55.259749 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.259789 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 13:12:55.260217 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.260099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37051 (local address 127.0.0.1:53938)
0928 13:12:55.260156 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.260184 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:37051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 13:12:55.260264 30886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260928 13:12:55.260346 30864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.189:37051: Network error: Client connection negotiation failed: client connection to 127.8.88.189:37051: connect: Connection refused (error 111)
I20260928 13:12:55.260473 30871 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:55.260555 30871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/wal
uuid: "d16f9adf05ca4b58861d3fffd8daa07c"
format_stamp: "Formatted at 2026-09-28 13:12:55 on dist-test-slave-p1lc"
server_key: "a33c074df6e6b40354710b850a50d237"
server_key_iv: "c091a4b5e537ac9ab4a2092c3a77b48c"
server_key_version: "encryptionkey@0"
I20260928 13:12:55.260614 30871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:55.282639 30871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:55.285115 30871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:55.285239 30871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:55.289126 30871 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.189:37051
I20260928 13:12:55.289175 30938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.189:37051 every 8 connection(s)
I20260928 13:12:55.289636 30871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/data/info.pb
I20260928 13:12:55.290803 30939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:55.292352 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 30871
I20260928 13:12:55.292464 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-1/wal/instance
I20260928 13:12:55.292670 8546 external_mini_cluster.cc:1428] Setting key 89162d67dccc9e297e5b21af207af81d
I20260928 13:12:55.292538 30939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601175, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.189@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.88.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.188@KRBTEST.COM" created.
I20260928 13:12:55.303301 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.293453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.8.88.190:38155)
0928 13:12:55.293523 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.293526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.294297 (+ 771us) server_negotiation.cc:407] Connection header received
0928 13:12:55.294817 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.294819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.294835 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.294934 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.295577 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.296162 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.297267 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.297482 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.297588 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.300097 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.300099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.300099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.300108 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.301690 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.302218 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.302220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.302220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.302283 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.302547 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.302548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.302549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.302713 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.302785 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.303109 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.303205 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 13:12:55.303490 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.293411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:38155 (local address 127.0.0.1:55394)
0928 13:12:55.294161 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.294198 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:55.294708 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:55.295012 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:55.295016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:55.295273 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:55.295471 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.295475 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.296257 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.296258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.297168 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.297171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.297605 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.297606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.297741 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.298288 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:55.298297 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:55.299994 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:55.301796 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.301806 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.301808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.302089 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.302353 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.302354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.302354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.302474 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.302791 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:55.302798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:55.302976 (+ 178us) client_negotiation.cc:769] Sending connection context
0928 13:12:55.303088 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 13:12:55.303251 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":87,"threads_started":1}
I20260928 13:12:55.304033 30939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601175, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.189@KRBTEST.COM for kudu/127.8.88.189@KRBTEST.COM
I20260928 13:12:55.311973 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.304315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37051 (local address 127.0.0.1:53948)
0928 13:12:55.304385 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.304396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:55.304436 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:55.304669 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:55.304670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:55.304813 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:55.304858 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.304859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.305744 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.305745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.306256 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.306257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.306766 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.306767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.306833 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.307477 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:55.307482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:55.309058 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:55.310845 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.310846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.310846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.311011 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.311231 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.311231 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.311232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.311286 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.311621 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:55.311622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:55.311668 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 13:12:55.311733 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 13:12:55.311835 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 13:12:55.312039 30944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.304363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.8.88.189:37051)
0928 13:12:55.304530 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.304535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.304558 (+ 23us) server_negotiation.cc:407] Connection header received
0928 13:12:55.304574 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.304576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.304586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.304668 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.304959 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.305637 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.306350 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.306675 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.306766 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.309141 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.309157 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.309157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.309164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.310743 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.311093 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.311096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.311096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.311161 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.311357 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.311358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.311358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.311552 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.311645 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.311761 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.311906 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260928 13:12:55.312791 30939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:12:55.313179 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.313056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:44179 (local address 127.0.0.1:34260)
0928 13:12:55.313105 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.313149 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 13:12:55.313310 30889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
I20260928 13:12:55.313771 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.313655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:44179 (local address 127.0.0.1:34272)
0928 13:12:55.313702 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.313747 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 13:12:55.313889 30939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:44179: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.188.keytab.
Entry for principal kudu/127.8.88.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.188.keytab.
I20260928 13:12:55.320554 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.188:44179
--webserver_interface=127.8.88.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.188.keytab
--principal=kudu/127.8.88.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:38155,127.8.88.189:37051,127.8.88.188:44179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:12:55.351526 30864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } attempt: 3
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601175, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for kudu/127.8.88.189@KRBTEST.COM
I20260928 13:12:55.359628 30944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.352125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53958 (local address 127.8.88.189:37051)
0928 13:12:55.352171 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.352173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.352184 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:12:55.352267 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.352268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.352282 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.352367 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.352744 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.353353 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.354211 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.354476 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.354582 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.356744 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.356746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.356747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.356757 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.358403 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.358806 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.358807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.358808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.358869 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.359073 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.359074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.359074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.359196 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.359272 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.359439 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.359521 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 13:12:55.359623 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.351990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37051 (local address 127.0.0.1:53958)
0928 13:12:55.352097 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.352117 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:55.352196 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:55.352375 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:55.352376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:55.352567 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:55.352640 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.352642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.353445 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.353446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.354024 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.354025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.354573 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.354574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.354671 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.355183 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:55.355187 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:55.356643 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:55.358527 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.358528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.358529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.358712 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.358942 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.358943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.358943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.359006 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.359282 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:55.359283 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:55.359330 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 13:12:55.359413 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 13:12:55.359519 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 13:12:55.360222 30864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:12:55.360582 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.360471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:44179 (local address 127.0.0.1:34288)
0928 13:12:55.360522 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.360549 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 13:12:55.360980 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.360897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:44179 (local address 127.0.0.1:34304)
0928 13:12:55.360927 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.360955 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260928 13:12:55.361085 30864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:44179: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
I20260928 13:12:55.363132 30939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } attempt: 1
I20260928 13:12:55.363508 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.363397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:44179 (local address 127.0.0.1:34310)
0928 13:12:55.363452 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.363480 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 13:12:55.363860 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.363763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:44179 (local address 127.0.0.1:34314)
0928 13:12:55.363809 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.363835 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 13:12:55.363970 30939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:44179: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
I20260928 13:12:55.392268 30864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } attempt: 1
I20260928 13:12:55.392786 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.392635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:44179 (local address 127.0.0.1:34316)
0928 13:12:55.392713 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.392755 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 13:12:55.393205 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.393085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:44179 (local address 127.0.0.1:34332)
0928 13:12:55.393153 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.393185 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 13:12:55.393299 30864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:44179: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
W20260928 13:12:55.406025 30946 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 13:12:55.406252 30946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:55.406270 30946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:55.406291 30946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:55.407528 30946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:55.407573 30946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:55.407588 30946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:55.407601 30946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:55.407613 30946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:55.409283 30946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:38155,127.8.88.189:37051,127.8.88.188:44179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.188.keytab
--principal=kudu/127.8.88.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.188:44179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.88.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:55.409530 30946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:55.409806 30946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 13:12:55.414808 30946 server_base.cc:1034] running on GCE node
W20260928 13:12:55.414781 30952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:55.414773 30951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:55.414773 30954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:55.415309 30946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:55.415683 30946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:55.416857 30946 hybrid_clock.cc:648] HybridClock initialized: now 1790601175416838 us; error 52 us; skew 500 ppm
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601175, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:55.419714 30946 init.cc:377] Logged in from keytab as kudu/127.8.88.188@KRBTEST.COM (short username kudu)
I20260928 13:12:55.420208 30946 webserver.cc:460] Webserver started at http://127.8.88.188:33867/ using document root <none> and password file <none>
I20260928 13:12:55.420415 30946 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:55.420455 30946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:55.420547 30946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:55.421643 30946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/data/instance:
uuid: "77db1b017f9c47cf85d19373626b315f"
format_stamp: "Formatted at 2026-09-28 13:12:55 on dist-test-slave-p1lc"
server_key: "e6d3c58a9f49692fc974d3702724604a"
server_key_iv: "1726991e9ebb7c59e5bd19052b600046"
server_key_version: "encryptionkey@0"
I20260928 13:12:55.421955 30946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/wal/instance:
uuid: "77db1b017f9c47cf85d19373626b315f"
format_stamp: "Formatted at 2026-09-28 13:12:55 on dist-test-slave-p1lc"
server_key: "e6d3c58a9f49692fc974d3702724604a"
server_key_iv: "1726991e9ebb7c59e5bd19052b600046"
server_key_version: "encryptionkey@0"
I20260928 13:12:55.423424 30946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 13:12:55.424196 30961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:55.424381 30946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:55.424475 30946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/wal
uuid: "77db1b017f9c47cf85d19373626b315f"
format_stamp: "Formatted at 2026-09-28 13:12:55 on dist-test-slave-p1lc"
server_key: "e6d3c58a9f49692fc974d3702724604a"
server_key_iv: "1726991e9ebb7c59e5bd19052b600046"
server_key_version: "encryptionkey@0"
I20260928 13:12:55.424536 30946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:55.432126 30939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } attempt: 2
I20260928 13:12:55.432654 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.432509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:44179 (local address 127.0.0.1:34338)
0928 13:12:55.432576 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.432623 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 13:12:55.433074 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.432989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:44179 (local address 127.0.0.1:34350)
0928 13:12:55.433033 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.433058 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 13:12:55.433256 30939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:44179: Network error: Client connection negotiation failed: client connection to 127.8.88.188:44179: connect: Connection refused (error 111)
I20260928 13:12:55.439533 30946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:55.441826 30946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:55.441993 30946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:55.446271 30946 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.188:44179
I20260928 13:12:55.446370 31013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.188:44179 every 8 connection(s)
I20260928 13:12:55.446727 30946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/data/info.pb
I20260928 13:12:55.447775 31014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:55.449493 31014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601175, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.188@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:55.455816 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 30946
I20260928 13:12:55.455945 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790600874459249-8546-0/minicluster-data/master-2/wal/instance
I20260928 13:12:55.456121 8546 external_mini_cluster.cc:1428] Setting key ccf9efa0b5634305e35ef95a0d0e4a60
I20260928 13:12:55.458949 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.450273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.8.88.190:38155)
0928 13:12:55.450339 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.450341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.450544 (+ 203us) server_negotiation.cc:407] Connection header received
0928 13:12:55.451017 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.451019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.451036 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.451096 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.451761 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.452275 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.453351 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.453544 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.453607 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.455972 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.455974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.455974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.455984 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.457565 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.457984 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.457986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.457986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.458050 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.458299 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.458300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.458300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.458416 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.458491 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.458775 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.458857 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 13:12:55.459162 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.450229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:38155 (local address 127.0.0.1:55398)
0928 13:12:55.450416 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.450467 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:55.450917 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:55.451176 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:55.451179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:55.451433 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:55.451634 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.451638 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.452368 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.452369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.453268 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.453270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.453642 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.453643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.453774 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.454365 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:55.454374 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:55.455847 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:55.457655 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.457662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.457665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.457893 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.458117 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.458118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.458118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.458227 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.458496 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:55.458503 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:55.458640 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 13:12:55.458753 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 13:12:55.458907 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260928 13:12:55.459852 31014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.188@KRBTEST.COM
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601175, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.188@KRBTEST.COM for kudu/127.8.88.189@KRBTEST.COM
I20260928 13:12:55.466064 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.456851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.8.88.188:44179)
0928 13:12:55.457139 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.457144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.457167 (+ 23us) server_negotiation.cc:407] Connection header received
0928 13:12:55.457227 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.457229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.457241 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.457330 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.457911 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.458595 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.459580 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.460011 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.460078 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.462756 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.462764 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.462765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.462772 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.464821 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.465223 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.465226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.465227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.465291 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.465491 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.465492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.465493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.465647 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.465816 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.465866 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.465944 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20260928 13:12:55.467051 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.460129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:37051 (local address 127.0.0.1:53960)
0928 13:12:55.460207 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.460219 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:55.460259 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:55.460438 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:55.460438 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:55.460600 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:55.460660 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.460661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.461350 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.461351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.461936 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.461938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.462462 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.462463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.462552 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.462978 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:55.462981 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:55.464176 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:55.465927 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.465928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.465929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.466179 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.466384 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.466385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.466386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.466445 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.466747 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:55.466748 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:55.466792 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 13:12:55.466860 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 13:12:55.466968 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 13:12:55.467316 30944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.460202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.8.88.189:37051)
0928 13:12:55.460245 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.460247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.460255 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:12:55.460346 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.460346 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.460357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.460416 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.460747 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.461277 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.462048 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.462364 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.462439 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.464263 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.464264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.464265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.464273 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.465820 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.466246 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.466248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.466248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.466306 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.466513 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.466514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.466515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.466671 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.466743 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.466908 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.467229 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:12:55.467638 31014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:12:55.468446 30864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } attempt: 2
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601175, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.188@KRBTEST.COM for kudu/127.8.88.188@KRBTEST.COM
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601175, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for kudu/127.8.88.188@KRBTEST.COM
I20260928 13:12:55.475378 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.467919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:44179 (local address 127.0.0.1:34360)
0928 13:12:55.467987 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.467996 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:55.468031 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:55.468183 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:55.468184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:55.468288 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:55.468316 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.468317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.469160 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.469161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.470032 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.470034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.470589 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.470590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.470638 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.471135 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:55.471140 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:55.472506 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:55.474343 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.474344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.474345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.474564 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.474781 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.474781 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.474782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.474840 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.475123 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:55.475124 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:55.475168 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 13:12:55.475240 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 13:12:55.475312 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 13:12:55.475512 31018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.467972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.8.88.188:44179)
0928 13:12:55.468088 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.468089 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.468094 (+ 5us) server_negotiation.cc:407] Connection header received
0928 13:12:55.468108 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.468109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.468128 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.468171 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.468388 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.469090 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.470160 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.470490 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.470779 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.472613 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.472614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.472614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.472620 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.474260 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.474641 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.474642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.474642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.474700 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.474914 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.474915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.474916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.475060 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.475132 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.475290 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.475370 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 13:12:55.476893 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.466985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55402 (local address 127.8.88.190:38155)
0928 13:12:55.467041 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.467043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.467051 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:12:55.467074 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.467075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.467085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.467136 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.467998 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.468762 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.469692 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.469919 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.469990 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.473319 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.473320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.473320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.473327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.475327 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.475706 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.475708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.475708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.475761 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.475975 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.475976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.475976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.476111 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.476260 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.476680 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.476759 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 13:12:55.477434 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.468705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:44179 (local address 127.0.0.1:34376)
0928 13:12:55.468751 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.468761 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:55.468798 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:55.469069 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:55.469070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:55.469236 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:55.469272 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.469274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.470068 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.470069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.471034 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.471037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.471594 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.471595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.471659 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.471958 (+ 299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:55.471963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:55.473673 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:55.476309 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.476310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.476311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.476519 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.476727 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.476728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.476728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.476784 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.477139 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:55.477140 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:55.477201 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 13:12:55.477254 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 13:12:55.477347 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 13:12:55.477414 31014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f: Bootstrap starting.
I20260928 13:12:55.478233 31014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:55.478262 31019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.468743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.8.88.188:44179)
0928 13:12:55.468941 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.468942 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.468951 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:12:55.468973 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.468973 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.468980 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.469051 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.469352 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.469948 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.471119 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.471479 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.471585 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.474171 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.474172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.474172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.474179 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.476230 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.476606 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.476607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.476607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.476665 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.476851 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.476852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.476852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.477041 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.477991 (+ 950us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.478031 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.478112 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":100,"threads_started":1}
I20260928 13:12:55.478550 31014 log.cc:824] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:55.479313 31014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f: No bootstrap required, opened a new log
I20260928 13:12:55.480394 30864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c: Bootstrap starting.
I20260928 13:12:55.480670 31014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } }
I20260928 13:12:55.480803 31014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77db1b017f9c47cf85d19373626b315f, State: Initialized, Role: FOLLOWER
I20260928 13:12:55.480861 31014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } }
I20260928 13:12:55.481140 30864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:55.481263 31014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:55.481216 31021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } } }
I20260928 13:12:55.481328 31021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:12:55.481451 30864 log.cc:824] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c: Log is configured to *not* fsync() on all Append() calls
W20260928 13:12:55.483108 31033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 13:12:55.483130 30864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c: No bootstrap required, opened a new log
W20260928 13:12:55.483162 31033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.189@KRBTEST.COM
I20260928 13:12:55.485278 30864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } }
I20260928 13:12:55.485414 30864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6eceea186d41ae905b4658d52ee66c, State: Initialized, Role: FOLLOWER
I20260928 13:12:55.485489 30864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } }
I20260928 13:12:55.485814 30864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:55.483369 30946 master_runner.cc:186] Error getting master registration for 127.8.88.190:38155: OK, instance_id { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" instance_seqno: 1790601175123309 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 13:12:55.486011 30946 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 13:12:55.487190 30944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.477777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53962 (local address 127.8.88.189:37051)
0928 13:12:55.477821 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.477823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.477877 (+ 54us) server_negotiation.cc:407] Connection header received
0928 13:12:55.477950 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.477951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.477959 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.478024 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.478652 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.479291 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.480320 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.480620 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.480665 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.483919 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.483920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.483921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.483928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.485814 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.486376 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.486377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.486378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.486438 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.486650 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.486651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.486652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.486778 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.486963 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.487001 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.487078 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:12:55.488649 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:55.489678 30838 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3b6eceea186d41ae905b4658d52ee66c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55402
I20260928 13:12:55.489805 30838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 13:12:55.489849 30838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:55.490834 30838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } }
I20260928 13:12:55.491078 30838 leader_election.cc:288] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [CANDIDATE]: Term 1 election: Requested vote from peers d16f9adf05ca4b58861d3fffd8daa07c (127.8.88.189:37051), 77db1b017f9c47cf85d19373626b315f (127.8.88.188:44179)
I20260928 13:12:55.491384 30988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b6eceea186d41ae905b4658d52ee66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "77db1b017f9c47cf85d19373626b315f"
I20260928 13:12:55.491504 30988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:55.488485 30796 master_runner.cc:186] Error getting master registration for 127.8.88.189:37051: OK, instance_id { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" instance_seqno: 1790601175287963 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 13:12:55.491761 30796 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 13:12:55.492483 30988 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eceea186d41ae905b4658d52ee66c in term 1.
I20260928 13:12:55.492800 30814 leader_election.cc:302] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6eceea186d41ae905b4658d52ee66c, 77db1b017f9c47cf85d19373626b315f; no voters:
I20260928 13:12:55.493418 31053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } } }
I20260928 13:12:55.493544 31053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:12:55.493604 31053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:55.493649 31053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [term 1 LEADER]: Becoming Leader. State: Replica: 3b6eceea186d41ae905b4658d52ee66c, State: Running, Role: LEADER
I20260928 13:12:55.493734 31053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } }
I20260928 13:12:55.497226 31035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b6eceea186d41ae905b4658d52ee66c. Latest consensus state: current_term: 1 leader_uuid: "3b6eceea186d41ae905b4658d52ee66c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } } }
I20260928 13:12:55.497337 31035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:55.500708 31056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:55.502174 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.492444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55408 (local address 127.8.88.190:38155)
0928 13:12:55.492486 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.492488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.493020 (+ 532us) server_negotiation.cc:407] Connection header received
0928 13:12:55.493054 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.493055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.493066 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.493125 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.493890 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.494663 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.495589 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.495905 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.495969 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.497813 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.497815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.497816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.497824 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.500228 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.500887 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.500888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.500888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.500949 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.501214 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.501215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.501215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.501351 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.501466 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.501568 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.501659 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:12:55.503980 31056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:55.505609 30988 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f [term 1 FOLLOWER]: Refusing update from remote peer 3b6eceea186d41ae905b4658d52ee66c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:55.505995 31035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:55.507200 30944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.492958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.8.88.189:37051)
0928 13:12:55.493016 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.493018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.494022 (+ 1004us) server_negotiation.cc:407] Connection header received
0928 13:12:55.494112 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.494114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.494125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.494153 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.495050 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.495788 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.497502 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.498301 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.498345 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.503465 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.503468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.503468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.503477 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.505813 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.506205 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.506207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.506207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.506268 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.506525 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.506526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.506527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.506674 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.506762 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.507020 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.507101 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 13:12:55.507982 31021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b6eceea186d41ae905b4658d52ee66c. Latest consensus state: current_term: 1 leader_uuid: "3b6eceea186d41ae905b4658d52ee66c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } } }
I20260928 13:12:55.508096 31021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:12:55.508334 31019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.492683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.8.88.188:44179)
0928 13:12:55.492733 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.492735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.493453 (+ 718us) server_negotiation.cc:407] Connection header received
0928 13:12:55.493803 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.493805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.493816 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.493849 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.496899 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.497639 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.500925 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.501295 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.501340 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.503394 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.503396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.503396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.503405 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.506593 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.507307 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.507308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.507309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.507377 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.507641 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.507642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.507642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.507776 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.507880 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.508159 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.508245 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 13:12:55.509822 31021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b6eceea186d41ae905b4658d52ee66c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } } }
I20260928 13:12:55.509907 31021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:12:55.509856 31053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b6eceea186d41ae905b4658d52ee66c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } } }
I20260928 13:12:55.509933 31053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:55.510238 31059 mvcc.cc:205] Tried to move back new op lower bound from 7334302414868611072 to 7334302414826885120. Current Snapshot: MvccSnapshot[applied={T|T < 7334302414868611072}]
I20260928 13:12:55.510805 31056 catalog_manager.cc:1347] Generated new cluster ID: a08a3fa1e38e4059a37ea9a470813f7f
I20260928 13:12:55.510854 31056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:55.522454 31056 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:55.523366 31056 catalog_manager.cc:1504] Loading token signing keys...
I20260928 13:12:55.524425 30939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } attempt: 3
Sep 28 13:12:55 dist-test-slave-p1lc krb5kdc[30783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601175, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.189@KRBTEST.COM for kudu/127.8.88.188@KRBTEST.COM
I20260928 13:12:55.533097 30941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.524721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:44179 (local address 127.0.0.1:34396)
0928 13:12:55.524772 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:55.524784 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:55.524821 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:55.524985 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:55.524986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:55.525185 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:55.525239 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.525241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.526116 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.526117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.526924 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:55.526927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.527523 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.527524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:55.527591 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.528105 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:55.528110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:55.529781 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:55.531829 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.531830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.531831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.532061 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.532321 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:55.532322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:55.532322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:55.532394 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:55.532732 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:55.532733 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:55.532796 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 13:12:55.532853 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 13:12:55.532978 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 13:12:55.533789 31019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:55.524750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34396 (local address 127.8.88.188:44179)
0928 13:12:55.524803 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.524805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.524814 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:12:55.524898 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.524899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.524908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.524982 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.525303 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.526029 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.526994 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:55.527336 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:55.527402 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:55.529884 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:55.529885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:55.529886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:55.529894 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:55.531717 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.532165 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.532166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.532167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.532240 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:55.532472 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:55.532473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:55.532473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:55.532624 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:55.532686 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:55.533614 (+ 928us) server_negotiation.cc:299] Negotiation successful
0928 13:12:55.533697 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 13:12:55.534889 31056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c: Generated new TSK 0
I20260928 13:12:55.535080 31056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:55.535542 30939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c: Bootstrap starting.
I20260928 13:12:55.536273 30939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:55.536553 30939 log.cc:824] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:55.540850 30939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c: No bootstrap required, opened a new log
I20260928 13:12:55.543001 30939 raft_consensus.cc:348] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } }
I20260928 13:12:55.543150 30939 raft_consensus.cc:729] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d16f9adf05ca4b58861d3fffd8daa07c, State: Initialized, Role: FOLLOWER
I20260928 13:12:55.543227 30939 consensus_queue.cc:260] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } }
I20260928 13:12:55.543540 30939 sys_catalog.cc:564] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:55.544996 30913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3b6eceea186d41ae905b4658d52ee66c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d16f9adf05ca4b58861d3fffd8daa07c"
I20260928 13:12:55.544910 30912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:55.546561 30912 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c [term 1 FOLLOWER]: Refusing update from remote peer 3b6eceea186d41ae905b4658d52ee66c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:55.547015 31053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 13:12:55.547382 31076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:12:55.547436 31076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:12:55.547869 30913 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6eceea186d41ae905b4658d52ee66c in term 1.
I20260928 13:12:55.549707 31082 sys_catalog.cc:455] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b6eceea186d41ae905b4658d52ee66c. Latest consensus state: current_term: 1 leader_uuid: "3b6eceea186d41ae905b4658d52ee66c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } } }
I20260928 13:12:55.549755 31065 sys_catalog.cc:455] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b6eceea186d41ae905b4658d52ee66c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } } }
I20260928 13:12:55.549853 31065 sys_catalog.cc:458] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:12:55.549808 31082 sys_catalog.cc:458] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:12:55.549923 31065 sys_catalog.cc:455] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3b6eceea186d41ae905b4658d52ee66c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } } }
I20260928 13:12:55.549980 31065 sys_catalog.cc:458] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c [sys.catalog]: This master's current role is: FOLLOWER
W20260928 13:12:55.568645 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:55.566925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53972 (local address 127.8.88.189:37051)
0928 13:12:55.566972 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.566974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.567925 (+ 951us) server_negotiation.cc:407] Connection header received
0928 13:12:55.568033 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.568034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.568046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.568079 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.568559 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53972: BlockingRecv error: recv got EOF from 127.0.0.1:53972 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 13:12:55.569617 31019 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:55.567347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.8.88.188:44179)
0928 13:12:55.567386 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.567388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.569067 (+ 1679us) server_negotiation.cc:407] Connection header received
0928 13:12:55.569182 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.569183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.569194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.569226 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.569538 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34400: BlockingRecv error: recv got EOF from 127.0.0.1:34400 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 13:12:55.570586 30868 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:55.567658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.8.88.190:38155)
0928 13:12:55.567699 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.567701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.569946 (+ 2245us) server_negotiation.cc:407] Connection header received
0928 13:12:55.570138 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.570139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.570152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.570186 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.570521 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55414: BlockingRecv error: recv got EOF from 127.0.0.1:55414 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 13:12:55.573320 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:55.572398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.8.88.189:37051)
0928 13:12:55.572436 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.572438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.572702 (+ 264us) server_negotiation.cc:407] Connection header received
0928 13:12:55.572813 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.572814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.572835 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.572917 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.573269 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53984: BlockingRecv error: recv got EOF from 127.0.0.1:53984 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 13:12:55.576532 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:55.575694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.8.88.189:37051)
0928 13:12:55.575730 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.575732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.575994 (+ 262us) server_negotiation.cc:407] Connection header received
0928 13:12:55.576093 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.576094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.576104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.576149 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.576480 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53986: BlockingRecv error: recv got EOF from 127.0.0.1:53986 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 13:12:55.580574 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:55.579768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.8.88.189:37051)
0928 13:12:55.579806 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.579808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.580044 (+ 236us) server_negotiation.cc:407] Connection header received
0928 13:12:55.580155 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.580156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.580167 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.580196 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.580522 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53998: BlockingRecv error: recv got EOF from 127.0.0.1:53998 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 13:12:55.586627 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:55.585781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.8.88.189:37051)
0928 13:12:55.585819 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.585820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.586066 (+ 246us) server_negotiation.cc:407] Connection header received
0928 13:12:55.586185 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.586186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.586197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.586227 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.586572 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54000: BlockingRecv error: recv got EOF from 127.0.0.1:54000 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 13:12:55.597214 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:55.596088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.8.88.189:37051)
0928 13:12:55.596166 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.596169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.596499 (+ 330us) server_negotiation.cc:407] Connection header received
0928 13:12:55.596614 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.596616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.596629 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.596661 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.597152 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54010: BlockingRecv error: recv got EOF from 127.0.0.1:54010 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 13:12:55.615835 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:55.614758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.8.88.189:37051)
0928 13:12:55.614811 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.614813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.615159 (+ 346us) server_negotiation.cc:407] Connection header received
0928 13:12:55.615287 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.615288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.615301 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.615333 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.615773 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54012: BlockingRecv error: recv got EOF from 127.0.0.1:54012 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 13:12:55.650377 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:55.649387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.8.88.189:37051)
0928 13:12:55.649435 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.649438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.649741 (+ 303us) server_negotiation.cc:407] Connection header received
0928 13:12:55.649852 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.649853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.649866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.649897 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.650314 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54018: BlockingRecv error: recv got EOF from 127.0.0.1:54018 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 13:12:55.716711 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:55.715981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54028 (local address 127.8.88.189:37051)
0928 13:12:55.716042 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.716044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.716101 (+ 57us) server_negotiation.cc:407] Connection header received
0928 13:12:55.716169 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.716170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.716182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.716268 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.716649 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54028: BlockingRecv error: recv got EOF from 127.0.0.1:54028 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 13:12:55.846602 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:55.845881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.8.88.189:37051)
0928 13:12:55.845954 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:55.845956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:55.846034 (+ 78us) server_negotiation.cc:407] Connection header received
0928 13:12:55.846082 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:55.846083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:55.846093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:55.846158 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:55.846524 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54042: BlockingRecv error: recv got EOF from 127.0.0.1:54042 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 13:12:55.960686 31035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3b6eceea186d41ae905b4658d52ee66c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6eceea186d41ae905b4658d52ee66c" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 38155 } } peers { permanent_uuid: "d16f9adf05ca4b58861d3fffd8daa07c" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 37051 } } peers { permanent_uuid: "77db1b017f9c47cf85d19373626b315f" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 44179 } } }
I20260928 13:12:55.960870 31035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6eceea186d41ae905b4658d52ee66c [sys.catalog]: This master's current role is: LEADER
W20260928 13:12:56.104903 30944 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:56.104175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54044 (local address 127.8.88.189:37051)
0928 13:12:56.104245 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:56.104247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:56.104257 (+ 10us) server_negotiation.cc:407] Connection header received
0928 13:12:56.104357 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:56.104358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:56.104370 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:56.104443 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:56.104818 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54044: BlockingRecv error: recv got EOF from 127.0.0.1:54044 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 13:12:56.483598 31033 catalog_manager.cc:1259] Loaded cluster ID: a08a3fa1e38e4059a37ea9a470813f7f
I20260928 13:12:56.483716 31033 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f: loading cluster ID for follower catalog manager: success
I20260928 13:12:56.484552 31033 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f: acquiring CA information for follower catalog manager: success
I20260928 13:12:56.484907 31033 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 77db1b017f9c47cf85d19373626b315f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:12:56.547899 31076 catalog_manager.cc:1259] Loaded cluster ID: a08a3fa1e38e4059a37ea9a470813f7f
I20260928 13:12:56.547976 31076 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c: loading cluster ID for follower catalog manager: success
I20260928 13:12:56.548797 31076 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c: acquiring CA information for follower catalog manager: success
I20260928 13:12:56.549091 31076 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d16f9adf05ca4b58861d3fffd8daa07c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:12:56.621289 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:56.618446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.8.88.189:37051)
0928 13:12:56.618567 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:56.618570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:56.618652 (+ 82us) server_negotiation.cc:407] Connection header received
0928 13:12:56.618766 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:56.618767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:56.618817 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:56.618921 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:56.619076 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:56.619586 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:56.620397 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:56.620604 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:56.620683 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:56.620914 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:56.621030 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:56.621126 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:56.621155 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 13:12:56.621179 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260928 13:12:56.625802 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:56.623396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.8.88.188:44179)
0928 13:12:56.623523 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:56.623525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:56.623540 (+ 15us) server_negotiation.cc:407] Connection header received
0928 13:12:56.623576 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:56.623577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:56.623613 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:56.623685 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:56.623779 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:56.624288 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:56.625133 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:56.625344 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:56.625401 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:56.625530 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:56.625633 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:56.625684 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:56.625720 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 13:12:56.625737 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":84,"threads_started":1}
I20260928 13:12:56.630414 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:56.627650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.8.88.189:37051)
0928 13:12:56.627694 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:56.627707 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:56.627992 (+ 285us) server_negotiation.cc:407] Connection header received
0928 13:12:56.628040 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:56.628041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:56.628055 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:56.628118 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:56.628322 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:56.629050 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:56.629791 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:56.630040 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:56.630096 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:56.630188 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:56.630218 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:56.630270 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:56.630302 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 13:12:56.630324 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:12:56.630926 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:56.627816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34422 (local address 127.8.88.188:44179)
0928 13:12:56.627855 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:56.627857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:56.628012 (+ 155us) server_negotiation.cc:407] Connection header received
0928 13:12:56.628184 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:56.628185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:56.628198 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:56.628233 (+ 35us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:56.628390 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:56.628855 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:56.630010 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:56.630311 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:56.630371 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:56.630640 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:56.630672 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:56.630698 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:56.630818 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 13:12:56.630843 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:12:56.631434 31160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:56.627727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55416 (local address 127.8.88.190:38155)
0928 13:12:56.627896 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:56.627898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:56.627906 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:12:56.627955 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:56.627956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:56.628008 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:56.628089 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:56.628181 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:56.629006 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:56.630410 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:56.630685 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:56.630756 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:56.630947 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:56.631096 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:56.631161 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:56.631306 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 13:12:56.631333 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":52,"threads_started":1}
I20260928 13:12:56.633163 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 30796
I20260928 13:12:56.640316 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 30871
I20260928 13:12:56.645093 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 30946
2026-09-28T13:12:56Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2847 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 13:12:56 dist-test-slave-p1lc krb5kdc[31165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 13:12:56 dist-test-slave-p1lc krb5kdc[31165](info): set up 2 sockets
Sep 28 13:12:56 dist-test-slave-p1lc krb5kdc[31165](info): commencing operation
krb5kdc: starting...
W20260928 13:12:57.721258 8546 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/test-user.keytab.
Sep 28 13:12:57 dist-test-slave-p1lc krb5kdc[31165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T13:12:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T13:12:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.88.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
Entry for principal kudu/127.8.88.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab.
I20260928 13:12:57.845155 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46023
--webserver_interface=127.8.88.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46023,127.8.88.189:33851,127.8.88.188:38185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 13:12:57.927881 31178 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 13:12:57.928167 31178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:57.928208 31178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:57.928267 31178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:57.929585 31178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:57.929651 31178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:57.929673 31178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:57.929694 31178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:57.929723 31178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:57.931306 31178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46023,127.8.88.189:33851,127.8.88.188:38185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.190.keytab
--principal=kudu/127.8.88.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.190:46023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.88.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:57.931612 31178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:57.931849 31178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:57.937206 31184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:57.937304 31178 server_base.cc:1034] running on GCE node
W20260928 13:12:57.937198 31186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:57.937215 31183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:57.937691 31178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:57.937984 31178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:57.939138 31178 hybrid_clock.cc:648] HybridClock initialized: now 1790601177939115 us; error 44 us; skew 500 ppm
Sep 28 13:12:57 dist-test-slave-p1lc krb5kdc[31165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601177, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:57.942081 31178 init.cc:377] Logged in from keytab as kudu/127.8.88.190@KRBTEST.COM (short username kudu)
I20260928 13:12:57.942592 31178 webserver.cc:460] Webserver started at http://127.8.88.190:38903/ using document root <none> and password file <none>
I20260928 13:12:57.942857 31178 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:57.942931 31178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:57.943085 31178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:57.944257 31178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/data/instance:
uuid: "20f83589d40b4ff0874f6b0bce185392"
format_stamp: "Formatted at 2026-09-28 13:12:57 on dist-test-slave-p1lc"
server_key: "38d6716205bb07fba42ab8d21a50c54d"
server_key_iv: "f7df799368fee9b6da6edf36cce79ef2"
server_key_version: "encryptionkey@0"
I20260928 13:12:57.944653 31178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/wal/instance:
uuid: "20f83589d40b4ff0874f6b0bce185392"
format_stamp: "Formatted at 2026-09-28 13:12:57 on dist-test-slave-p1lc"
server_key: "38d6716205bb07fba42ab8d21a50c54d"
server_key_iv: "f7df799368fee9b6da6edf36cce79ef2"
server_key_version: "encryptionkey@0"
I20260928 13:12:57.946168 31178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 13:12:57.946967 31193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:57.947150 31178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 13:12:57.947265 31178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/wal
uuid: "20f83589d40b4ff0874f6b0bce185392"
format_stamp: "Formatted at 2026-09-28 13:12:57 on dist-test-slave-p1lc"
server_key: "38d6716205bb07fba42ab8d21a50c54d"
server_key_iv: "f7df799368fee9b6da6edf36cce79ef2"
server_key_version: "encryptionkey@0"
I20260928 13:12:57.947345 31178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:57.964028 31178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:57.966320 31178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:57.966507 31178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:57.970995 31178 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.190:46023
I20260928 13:12:57.971036 31245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.190:46023 every 8 connection(s)
I20260928 13:12:57.971432 31178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/data/info.pb
I20260928 13:12:57.972464 31246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:57.974117 31246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:12:57.980428 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 31178
I20260928 13:12:57.980563 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-0/wal/instance
Sep 28 13:12:57 dist-test-slave-p1lc krb5kdc[31165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601177, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:57.980814 8546 external_mini_cluster.cc:1428] Setting key 12fc5b482f912dd18e0092f8307aef67
I20260928 13:12:57.984865 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:57.974884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.8.88.190:46023)
0928 13:12:57.975037 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:57.975045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:57.975205 (+ 160us) server_negotiation.cc:407] Connection header received
0928 13:12:57.975726 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:57.975728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:57.975746 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:57.975838 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:57.976416 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:57.977344 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:57.978298 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:57.978624 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:57.978714 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:57.981224 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:57.981231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:57.981232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:57.981239 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:57.982914 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:57.983428 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:57.983431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:57.983431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:57.983520 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:57.983811 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:57.983812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:57.983812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:57.984046 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:57.984155 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:57.984468 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 13:12:57.984686 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260928 13:12:57.985126 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:57.974839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46023 (local address 127.0.0.1:37282)
0928 13:12:57.975019 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:57.975063 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:57.975569 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:57.975870 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:57.975872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:57.976140 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:57.976317 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:57.976322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:57.977446 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:57.977447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:57.978195 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:57.978198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:57.978745 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:57.978747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:57.978810 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:57.979324 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:57.979332 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:57.981115 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:57.983010 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:57.983018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:57.983021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:57.983321 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:57.983600 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:57.983602 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:57.983602 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:57.983720 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:57.984141 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:57.984148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:57.984231 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 13:12:57.984394 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 13:12:57.984610 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260928 13:12:57.986006 31246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:12:57.986411 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:57.986274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:33851 (local address 127.0.0.1:59740)
0928 13:12:57.986326 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:57.986380 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 13:12:57.986568 31194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.88.189:33851: connect: Connection refused (error 111)
I20260928 13:12:57.987016 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:57.986904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:33851 (local address 127.0.0.1:59756)
0928 13:12:57.986955 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:57.986986 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 13:12:57.987241 31246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.189:33851: Network error: Client connection negotiation failed: client connection to 127.8.88.189:33851: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.8.88.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.189.keytab.
Entry for principal kudu/127.8.88.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.189.keytab.
I20260928 13:12:58.008836 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.189:33851
--webserver_interface=127.8.88.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.189.keytab
--principal=kudu/127.8.88.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46023,127.8.88.189:33851,127.8.88.188:38185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:12:58.036484 31246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } attempt: 1
I20260928 13:12:58.037042 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.036857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:33851 (local address 127.0.0.1:59766)
0928 13:12:58.036952 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.037002 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 13:12:58.037480 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.037387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:33851 (local address 127.0.0.1:59770)
0928 13:12:58.037429 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.037459 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 13:12:58.037612 31246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.189:33851: Network error: Client connection negotiation failed: client connection to 127.8.88.189:33851: connect: Connection refused (error 111)
W20260928 13:12:58.090801 31253 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 13:12:58.091025 31253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:58.091043 31253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:58.091063 31253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:58.092255 31253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:58.092303 31253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:58.092316 31253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:58.092329 31253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:58.092341 31253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:58.093959 31253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46023,127.8.88.189:33851,127.8.88.188:38185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.189.keytab
--principal=kudu/127.8.88.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.189:33851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.88.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:58.094231 31253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:58.094470 31253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:58.099494 31258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:58.099509 31259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:58.099543 31253 server_base.cc:1034] running on GCE node
W20260928 13:12:58.099509 31261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:58.100023 31253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:58.100338 31253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:58.101524 31253 hybrid_clock.cc:648] HybridClock initialized: now 1790601178101512 us; error 42 us; skew 500 ppm
Sep 28 13:12:58 dist-test-slave-p1lc krb5kdc[31165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601178, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:58.104359 31253 init.cc:377] Logged in from keytab as kudu/127.8.88.189@KRBTEST.COM (short username kudu)
I20260928 13:12:58.104990 31253 webserver.cc:460] Webserver started at http://127.8.88.189:42053/ using document root <none> and password file <none>
I20260928 13:12:58.105226 31253 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:58.105293 31253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:58.105448 31253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:58.105753 31246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } attempt: 2
I20260928 13:12:58.106282 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.106137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:33851 (local address 127.0.0.1:59778)
0928 13:12:58.106205 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.106245 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 13:12:58.106626 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.106527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:33851 (local address 127.0.0.1:59782)
0928 13:12:58.106574 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.106604 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.189:33851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 13:12:58.106721 31246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.189:33851: Network error: Client connection negotiation failed: client connection to 127.8.88.189:33851: connect: Connection refused (error 111)
I20260928 13:12:58.106935 31253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/data/instance:
uuid: "760325b0f90f4a6c870a56835dcca3f6"
format_stamp: "Formatted at 2026-09-28 13:12:58 on dist-test-slave-p1lc"
server_key: "168065bcf2e02cfdb7e7014d49779586"
server_key_iv: "8bc1460576e23615b4a17de1eed0dab3"
server_key_version: "encryptionkey@0"
I20260928 13:12:58.107304 31253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/wal/instance:
uuid: "760325b0f90f4a6c870a56835dcca3f6"
format_stamp: "Formatted at 2026-09-28 13:12:58 on dist-test-slave-p1lc"
server_key: "168065bcf2e02cfdb7e7014d49779586"
server_key_iv: "8bc1460576e23615b4a17de1eed0dab3"
server_key_version: "encryptionkey@0"
I20260928 13:12:58.108747 31253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 13:12:58.109789 31268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:58.109970 31253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 13:12:58.110069 31253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/wal
uuid: "760325b0f90f4a6c870a56835dcca3f6"
format_stamp: "Formatted at 2026-09-28 13:12:58 on dist-test-slave-p1lc"
server_key: "168065bcf2e02cfdb7e7014d49779586"
server_key_iv: "8bc1460576e23615b4a17de1eed0dab3"
server_key_version: "encryptionkey@0"
I20260928 13:12:58.110148 31253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:58.124716 31253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:58.126914 31253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:58.127034 31253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:58.131135 31253 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.189:33851
I20260928 13:12:58.131191 31320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.189:33851 every 8 connection(s)
I20260928 13:12:58.131541 31253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/data/info.pb
I20260928 13:12:58.132622 31321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:58.134161 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 31253
I20260928 13:12:58.134262 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-1/wal/instance
I20260928 13:12:58.134487 8546 external_mini_cluster.cc:1428] Setting key 3caa4f96d8ca06d79dcd2b67635dbfac
I20260928 13:12:58.134675 31321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:12:58 dist-test-slave-p1lc krb5kdc[31165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601178, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.189@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.88.188@KRBTEST.COM; defaulting to no policy
I20260928 13:12:58.145313 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.135425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.8.88.190:46023)
0928 13:12:58.135496 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.135499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.135792 (+ 293us) server_negotiation.cc:407] Connection header received
0928 13:12:58.136288 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.136289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.136306 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.136379 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.137235 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.137733 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.139376 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.139596 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.139660 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.142255 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.142257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.142257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.142270 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.143763 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.144212 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.144213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.144214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.144282 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.144543 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.144545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.144545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.144688 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.144796 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.145123 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.145212 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.88.188@KRBTEST.COM" created.
I20260928 13:12:58.145535 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.135404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46023 (local address 127.0.0.1:37288)
0928 13:12:58.135626 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.135684 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:58.136191 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:58.136449 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:58.136452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:58.136825 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:58.137143 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.137149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.137827 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.137829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.139274 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.139277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.139697 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.139699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.139886 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.140451 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:58.140463 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:58.142132 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:58.143877 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.143885 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.143888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.144133 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.144361 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.144361 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.144362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.144478 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.144790 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:58.144797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:58.144988 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 13:12:58.145101 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 13:12:58.145283 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":101,"threads_started":1}
I20260928 13:12:58.146092 31321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 13:12:58 dist-test-slave-p1lc krb5kdc[31165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601178, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.189@KRBTEST.COM for kudu/127.8.88.189@KRBTEST.COM
I20260928 13:12:58.155344 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.146389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:33851 (local address 127.0.0.1:59786)
0928 13:12:58.146597 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.146608 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:58.146660 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:58.146842 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:58.146844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:58.147012 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:58.147068 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.147069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.148265 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.148266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.149175 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.149178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.149814 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.149816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.149906 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.150480 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:58.150486 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:58.151989 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:58.154250 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.154251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.154251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.154418 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.154606 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.154607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.154607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.154660 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.155038 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:58.155039 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:58.155088 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 13:12:58.155146 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 13:12:58.155270 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260928 13:12:58.155468 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.146446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.8.88.189:33851)
0928 13:12:58.146608 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.146613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.146657 (+ 44us) server_negotiation.cc:407] Connection header received
0928 13:12:58.146723 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.146726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.146740 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.146816 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.147139 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.148176 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.149299 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.149705 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.149770 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.152079 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.152091 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.152091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.152099 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.154151 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.154491 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.154494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.154494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.154549 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.154733 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.154735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.154735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.154941 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.155028 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.155189 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.155325 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260928 13:12:58.156103 31321 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:12:58.156466 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.156345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:38185 (local address 127.0.0.1:37804)
0928 13:12:58.156403 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.156445 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 13:12:58.156594 31270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
I20260928 13:12:58.157063 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.156976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:38185 (local address 127.0.0.1:37808)
0928 13:12:58.157017 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.157046 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 13:12:58.157164 31321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:38185: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.88.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.188.keytab.
Entry for principal kudu/127.8.88.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.188.keytab.
I20260928 13:12:58.164108 8546 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
/tmp/dist-test-taskcZlb_Q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.188:38185
--webserver_interface=127.8.88.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.188.keytab
--principal=kudu/127.8.88.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.88.148:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.88.190:46023,127.8.88.189:33851,127.8.88.188:38185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 13:12:58.197856 31246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } attempt: 3
Sep 28 13:12:58 dist-test-slave-p1lc krb5kdc[31165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601177, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for kudu/127.8.88.189@KRBTEST.COM
I20260928 13:12:58.205102 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.198376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.8.88.189:33851)
0928 13:12:58.198440 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.198442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.198453 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:12:58.198512 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.198513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.198524 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.198591 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.198939 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.199461 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.200244 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.200485 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.200570 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.202607 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.202608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.202608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.202617 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.203955 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.204300 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.204301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.204302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.204361 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.204550 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.204551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.204551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.204674 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.204745 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.204917 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.205004 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 13:12:58.205111 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.198311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:33851 (local address 127.0.0.1:59800)
0928 13:12:58.198371 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.198382 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:58.198449 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:58.198598 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:58.198598 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:58.198766 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:58.198836 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.198838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.199555 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.199556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.200158 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.200161 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.200567 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.200568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.200651 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.201201 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:58.201205 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:58.202480 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:58.204043 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.204044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.204044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.204230 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.204426 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.204426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.204427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.204490 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.204736 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:58.204737 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:58.204786 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 13:12:58.204869 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 13:12:58.205004 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 13:12:58.206382 31321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } attempt: 1
I20260928 13:12:58.206772 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.206651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:38185 (local address 127.0.0.1:37816)
0928 13:12:58.206703 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.206738 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 13:12:58.207095 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.207016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:38185 (local address 127.0.0.1:37832)
0928 13:12:58.207051 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.207073 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 13:12:58.207100 31246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } has no permanent_uuid. Determining permanent_uuid...
W20260928 13:12:58.207239 31321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:38185: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
I20260928 13:12:58.207422 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.207316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:38185 (local address 127.0.0.1:37846)
0928 13:12:58.207354 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.207391 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260928 13:12:58.207756 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.207661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:38185 (local address 127.0.0.1:37860)
0928 13:12:58.207710 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.207738 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 13:12:58.207986 31246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:38185: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
I20260928 13:12:58.239169 31246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } attempt: 1
I20260928 13:12:58.239727 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.239536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:38185 (local address 127.0.0.1:37868)
0928 13:12:58.239607 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.239688 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 13:12:58.240270 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.240161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:38185 (local address 127.0.0.1:37872)
0928 13:12:58.240210 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.240243 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 13:12:58.240402 31246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:38185: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
W20260928 13:12:58.248369 31328 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 13:12:58.248594 31328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 13:12:58.248612 31328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 13:12:58.248665 31328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 13:12:58.249904 31328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 13:12:58.249957 31328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 13:12:58.249971 31328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 13:12:58.249984 31328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 13:12:58.249996 31328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 13:12:58.251673 31328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.88.148:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.88.190:46023,127.8.88.189:33851,127.8.88.188:38185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/krb5kdc/kudu_127.8.88.188.keytab
--principal=kudu/127.8.88.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.88.188:38185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.88.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 12:08:40 UTC on ccycloud.quasar-tlwibe.root.comops.site
build id 4677203
I20260928 13:12:58.251940 31328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 13:12:58.252192 31328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 13:12:58.257294 31336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:58.257377 31328 server_base.cc:1034] running on GCE node
W20260928 13:12:58.257273 31334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 13:12:58.257278 31333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 13:12:58.257822 31328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 13:12:58.258193 31328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 13:12:58.259367 31328 hybrid_clock.cc:648] HybridClock initialized: now 1790601178259370 us; error 42 us; skew 500 ppm
Sep 28 13:12:58 dist-test-slave-p1lc krb5kdc[31165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601178, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 13:12:58.262091 31328 init.cc:377] Logged in from keytab as kudu/127.8.88.188@KRBTEST.COM (short username kudu)
I20260928 13:12:58.262665 31328 webserver.cc:460] Webserver started at http://127.8.88.188:40941/ using document root <none> and password file <none>
I20260928 13:12:58.262920 31328 fs_manager.cc:359] Metadata directory not provided
I20260928 13:12:58.262989 31328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 13:12:58.263130 31328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 13:12:58.264221 31328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/data/instance:
uuid: "ad09d25b31134ee3bdc65149fc0cef63"
format_stamp: "Formatted at 2026-09-28 13:12:58 on dist-test-slave-p1lc"
server_key: "6c218db71911d573b29436b67f7ddbe6"
server_key_iv: "99416e06cc31dd4a04c0cbd146b4cc35"
server_key_version: "encryptionkey@0"
I20260928 13:12:58.264575 31328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/wal/instance:
uuid: "ad09d25b31134ee3bdc65149fc0cef63"
format_stamp: "Formatted at 2026-09-28 13:12:58 on dist-test-slave-p1lc"
server_key: "6c218db71911d573b29436b67f7ddbe6"
server_key_iv: "99416e06cc31dd4a04c0cbd146b4cc35"
server_key_version: "encryptionkey@0"
I20260928 13:12:58.266000 31328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 13:12:58.266927 31343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 13:12:58.267318 31328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 13:12:58.267485 31328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/data,/tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/wal
uuid: "ad09d25b31134ee3bdc65149fc0cef63"
format_stamp: "Formatted at 2026-09-28 13:12:58 on dist-test-slave-p1lc"
server_key: "6c218db71911d573b29436b67f7ddbe6"
server_key_iv: "99416e06cc31dd4a04c0cbd146b4cc35"
server_key_version: "encryptionkey@0"
I20260928 13:12:58.267570 31328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 13:12:58.275398 31321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } attempt: 2
I20260928 13:12:58.275847 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.275700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:38185 (local address 127.0.0.1:37874)
0928 13:12:58.275767 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.275812 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 13:12:58.276325 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.276193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:38185 (local address 127.0.0.1:37882)
0928 13:12:58.276272 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.276302 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 13:12:58.276439 31321 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.88.188:38185: Network error: Client connection negotiation failed: client connection to 127.8.88.188:38185: connect: Connection refused (error 111)
I20260928 13:12:58.277176 31328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 13:12:58.279479 31328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 13:12:58.279637 31328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 13:12:58.283586 31328 rpc_server.cc:307] RPC server started. Bound to: 127.8.88.188:38185
I20260928 13:12:58.283620 31395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.88.188:38185 every 8 connection(s)
I20260928 13:12:58.284014 31328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/data/info.pb
I20260928 13:12:58.285110 31396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 13:12:58.286788 31396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:12:58.289381 8546 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu as pid 31328
I20260928 13:12:58.289511 8546 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcZlb_Q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790600874459249-8546-0/minicluster-data/master-2/wal/instance
I20260928 13:12:58.289772 8546 external_mini_cluster.cc:1428] Setting key 460ba79d333bff5998be1c9c5557f1cc
Sep 28 13:12:58 dist-test-slave-p1lc krb5kdc[31165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601178, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.188@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
Sep 28 13:12:58 dist-test-slave-p1lc krb5kdc[31165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.188@KRBTEST.COM
I20260928 13:12:58.298062 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.287526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.8.88.190:46023)
0928 13:12:58.287594 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.287597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.287898 (+ 301us) server_negotiation.cc:407] Connection header received
0928 13:12:58.288363 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.288364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.288374 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.288556 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.289573 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.290294 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.292025 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.292383 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.292458 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.294886 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.294888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.294888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.294897 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.296579 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.297059 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.297061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.297062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.297131 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.297390 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.297391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.297391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.297518 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.297590 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.297878 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.297968 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 13:12:58.298257 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.287517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.190:46023 (local address 127.0.0.1:37302)
0928 13:12:58.287758 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.287798 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:58.288277 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:58.288728 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:58.288736 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:58.289155 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:58.289450 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.289457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.290412 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.290413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.291704 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.291708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.292492 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.292493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.292616 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.293229 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:58.293239 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:58.294791 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:58.296660 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.296668 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.296670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.296985 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.297202 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.297203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.297203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.297315 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.297581 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:58.297588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:58.297739 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 13:12:58.297842 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 13:12:58.298006 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":101,"threads_started":1}
I20260928 13:12:58.298875 31396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:12:58.299674 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.290131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.8.88.188:38185)
0928 13:12:58.290320 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.290326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.290481 (+ 155us) server_negotiation.cc:407] Connection header received
0928 13:12:58.290508 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.290510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.290532 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.290621 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.291414 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.292595 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.293387 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.293708 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.293798 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.296380 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.296388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.296389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.296395 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.297969 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.298558 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.298561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.298561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.298627 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.298828 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.298829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.298829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.299000 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.299144 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.299433 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.299555 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
Sep 28 13:12:58 dist-test-slave-p1lc krb5kdc[31165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601178, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.188@KRBTEST.COM for kudu/127.8.88.189@KRBTEST.COM
Sep 28 13:12:58 dist-test-slave-p1lc krb5kdc[31165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.190@KRBTEST.COM
I20260928 13:12:58.306581 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.299139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.189:33851 (local address 127.0.0.1:59804)
0928 13:12:58.299186 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.299196 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:58.299231 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:58.299398 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:58.299398 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:58.299584 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:58.299645 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.299648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.300552 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.300553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.301348 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.301351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.301952 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.301953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.302012 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.302468 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:58.302472 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:58.303792 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:58.305374 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.305375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.305376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.305601 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.305838 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.305839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.305839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.305906 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.306191 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:58.306192 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:58.306248 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 13:12:58.306324 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 13:12:58.306465 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 13:12:58.306577 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.299213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.8.88.189:33851)
0928 13:12:58.299285 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.299287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.299294 (+ 7us) server_negotiation.cc:407] Connection header received
0928 13:12:58.299319 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.299321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.299332 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.299389 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.299736 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.300459 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.301460 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.301869 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.301949 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.303887 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.303888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.303888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.303896 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.305285 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.305691 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.305692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.305692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.305756 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.305966 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.305967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.305967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.306117 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.306181 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.306374 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.306464 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 13:12:58.307063 31396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } has no permanent_uuid. Determining permanent_uuid...
I20260928 13:12:58.308794 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.300749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37304 (local address 127.8.88.190:46023)
0928 13:12:58.300814 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.300816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.300824 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:12:58.300845 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.300846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.300857 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.300935 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.301665 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.302263 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.303014 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.303256 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.303314 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.305828 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.305829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.305830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.305838 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.307685 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.308027 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.308028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.308028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.308093 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.308310 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.308311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.308312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.308444 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.308626 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.308661 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.308729 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 13:12:58 dist-test-slave-p1lc krb5kdc[31165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601178, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.188@KRBTEST.COM for kudu/127.8.88.188@KRBTEST.COM
Sep 28 13:12:58 dist-test-slave-p1lc krb5kdc[31165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.88.189@KRBTEST.COM
I20260928 13:12:58.314631 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.307347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:38185 (local address 127.0.0.1:37902)
0928 13:12:58.307382 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.307393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:58.307431 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:58.307560 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:58.307560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:58.307677 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:58.307731 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.307733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.308551 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.308552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.309302 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.309305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.309961 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.309962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.310003 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.310408 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:58.310413 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:58.311548 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:58.313404 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.313406 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.313407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.313638 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.313919 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.313920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.313921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.313997 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.314269 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:58.314270 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:58.314318 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 13:12:58.314388 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 13:12:58.314480 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 13:12:58.314913 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.307398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.8.88.188:38185)
0928 13:12:58.307438 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.307440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.307452 (+ 12us) server_negotiation.cc:407] Connection header received
0928 13:12:58.307475 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.307476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.307489 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.307559 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.307802 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.308477 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.309380 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.309888 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.309966 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.311632 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.311634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.311634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.311640 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.313293 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.313768 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.313769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.313769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.313826 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.314069 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.314069 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.314070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.314197 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.314258 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.314758 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.314835 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 13:12:58.315529 31246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } attempt: 2
I20260928 13:12:58.316481 31396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63: Bootstrap starting.
I20260928 13:12:58.317375 31396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:58.317611 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.309526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.8.88.189:33851)
0928 13:12:58.309572 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.309574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.309594 (+ 20us) server_negotiation.cc:407] Connection header received
0928 13:12:58.309614 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.309615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.309624 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.309671 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.310279 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.310729 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.311398 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.311719 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.311771 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.314855 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.314856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.314856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.314864 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.316464 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.316827 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.316828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.316829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.316909 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.317101 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.317102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.317102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.317217 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.317384 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.317420 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.317499 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 13:12:58.317685 31396 log.cc:824] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:58.318701 31396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63: No bootstrap required, opened a new log
I20260928 13:12:58.319010 8546 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 13:12:58.320451 31396 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } }
I20260928 13:12:58.320640 31396 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad09d25b31134ee3bdc65149fc0cef63, State: Initialized, Role: FOLLOWER
I20260928 13:12:58.319911 31220 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "20f83589d40b4ff0874f6b0bce185392"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37304
I20260928 13:12:58.321087 31396 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } }
I20260928 13:12:58.321427 31402 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } } }
I20260928 13:12:58.321502 31396 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:58.321518 31402 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 13:12:58 dist-test-slave-p1lc krb5kdc[31165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601177, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.190@KRBTEST.COM for kudu/127.8.88.188@KRBTEST.COM
W20260928 13:12:58.323426 31413 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:12:58.323490 31413 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:12:58.325666 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.315796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:38185 (local address 127.0.0.1:37910)
0928 13:12:58.315848 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.315858 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:58.315894 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:58.318163 (+ 2269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:58.318165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:58.318337 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:58.318391 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.318393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.319252 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.319254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.320029 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.320032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.320518 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.320519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.320581 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.321087 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:58.321092 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:58.322700 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:58.324474 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.324476 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.324477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.324702 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.324982 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.324983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.324983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.325051 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.325345 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:58.325346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:58.325394 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 13:12:58.325461 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 13:12:58.325569 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 13:12:58.325664 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.315828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.8.88.188:38185)
0928 13:12:58.317994 (+ 2166us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.317997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.318006 (+ 9us) server_negotiation.cc:407] Connection header received
0928 13:12:58.318025 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.318026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.318036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.318471 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.318503 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.319166 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.320136 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.320430 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.320511 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.322826 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.322828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.322829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.322837 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.324410 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.324822 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.324824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.324824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.324902 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.325114 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.325115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.325115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.325265 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.325339 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.325496 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.325575 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2145}
I20260928 13:12:58.324158 31328 master_runner.cc:186] Error getting master registration for 127.8.88.190:46023: OK, instance_id { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" instance_seqno: 1790601177969698 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 13:12:58.327221 31328 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 13:12:58.327864 31246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392: Bootstrap starting.
I20260928 13:12:58.328464 31246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:58.328754 31246 log.cc:824] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:58.329602 31246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392: No bootstrap required, opened a new log
I20260928 13:12:58.330638 31246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } }
I20260928 13:12:58.330775 31246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f83589d40b4ff0874f6b0bce185392, State: Initialized, Role: FOLLOWER
I20260928 13:12:58.330868 31246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } }
I20260928 13:12:58.331164 31416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } } }
I20260928 13:12:58.331230 31246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:58.331257 31220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 13:12:58.331238 31416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:12:58.331350 31220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:58.332221 31220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } }
I20260928 13:12:58.332489 31220 leader_election.cc:288] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [CANDIDATE]: Term 1 election: Requested vote from peers 760325b0f90f4a6c870a56835dcca3f6 (127.8.88.189:33851), ad09d25b31134ee3bdc65149fc0cef63 (127.8.88.188:38185)
I20260928 13:12:58.332854 31370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20f83589d40b4ff0874f6b0bce185392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ad09d25b31134ee3bdc65149fc0cef63"
I20260928 13:12:58.333019 31370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:58.334218 31370 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f83589d40b4ff0874f6b0bce185392 in term 1.
I20260928 13:12:58.334501 31195 leader_election.cc:302] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20f83589d40b4ff0874f6b0bce185392, ad09d25b31134ee3bdc65149fc0cef63; no voters:
I20260928 13:12:58.334646 31416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 13:12:58.334802 31416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [term 1 LEADER]: Becoming Leader. State: Replica: 20f83589d40b4ff0874f6b0bce185392, State: Running, Role: LEADER
I20260928 13:12:58.334910 31416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } }
I20260928 13:12:58.335371 31416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20f83589d40b4ff0874f6b0bce185392. Latest consensus state: current_term: 1 leader_uuid: "20f83589d40b4ff0874f6b0bce185392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } } }
I20260928 13:12:58.335449 31416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:58.335920 31436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 13:12:58.336432 31436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 13:12:58.336966 31178 master_runner.cc:186] Error getting master registration for 127.8.88.189:33851: OK, instance_id { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" instance_seqno: 1790601178129978 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 13:12:58.339818 31178 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 13:12:58.341041 31370 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63 [term 1 FOLLOWER]: Refusing update from remote peer 20f83589d40b4ff0874f6b0bce185392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:58.341427 31434 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:58.343345 31402 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20f83589d40b4ff0874f6b0bce185392. Latest consensus state: current_term: 1 leader_uuid: "20f83589d40b4ff0874f6b0bce185392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } } }
I20260928 13:12:58.343443 31402 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:12:58.343467 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.333935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.8.88.189:33851)
0928 13:12:58.333977 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.333979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.334545 (+ 566us) server_negotiation.cc:407] Connection header received
0928 13:12:58.334646 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.334648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.334659 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.334692 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.335587 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.336302 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.337336 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.338140 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.338192 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.339770 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.339771 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.339771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.339779 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.342074 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.342475 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.342476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.342476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.342538 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.342777 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.342778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.342778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.342910 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.342986 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.343278 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.343373 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:12:58.351024 31434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20f83589d40b4ff0874f6b0bce185392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } } }
I20260928 13:12:58.351176 31434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [sys.catalog]: This master's current role is: LEADER
I20260928 13:12:58.351713 31402 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20f83589d40b4ff0874f6b0bce185392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } } }
I20260928 13:12:58.351830 31402 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:12:58.352043 31249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.333846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.8.88.190:46023)
0928 13:12:58.333889 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.333891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.337334 (+ 3443us) server_negotiation.cc:407] Connection header received
0928 13:12:58.337439 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.337440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.337450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.337477 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.338767 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.339552 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.340550 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.341145 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.341227 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.346786 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.346788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.346788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.346796 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.348882 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.350048 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.350050 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.350050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.350120 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.350455 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.350456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.350456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.350592 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.350664 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.351018 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.351106 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 13:12:58.352248 31439 mvcc.cc:205] Tried to move back new op lower bound from 7334302426474455040 to 7334302426460585984. Current Snapshot: MvccSnapshot[applied={T|T < 7334302426474455040 or (T in {7334302426474455040})}]
I20260928 13:12:58.352293 31436 catalog_manager.cc:1347] Generated new cluster ID: 08156af25934493e9c49e711d3e9ec65
I20260928 13:12:58.352327 31436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 13:12:58.361644 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.334339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.8.88.188:38185)
0928 13:12:58.334380 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.334382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.343858 (+ 9476us) server_negotiation.cc:407] Connection header received
0928 13:12:58.344112 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.344114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.344127 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.344158 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.344978 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.345710 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.346712 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.346986 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.347028 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.348943 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.348944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.348945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.348953 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.350611 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.351355 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.351368 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.351368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.351430 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.351998 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.352000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.352000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.352192 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.352298 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.361409 (+ 9111us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.361532 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 13:12:58.367594 31321 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } attempt: 3
I20260928 13:12:58.373361 31436 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 13:12:58.374150 31436 catalog_manager.cc:1504] Loading token signing keys...
Sep 28 13:12:58 dist-test-slave-p1lc krb5kdc[31165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790601178, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.88.189@KRBTEST.COM for kudu/127.8.88.188@KRBTEST.COM
I20260928 13:12:58.382766 31436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392: Generated new TSK 0
I20260928 13:12:58.382972 31436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 13:12:58.383841 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.367964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.88.188:38185 (local address 127.0.0.1:37928)
0928 13:12:58.368108 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 13:12:58.368120 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 13:12:58.368164 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 13:12:58.368550 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 13:12:58.368552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 13:12:58.368748 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 13:12:58.368800 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.368803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.369815 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.369816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.370665 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 13:12:58.370668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.371312 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.371313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 13:12:58.371395 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.378407 (+ 7012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 13:12:58.378413 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 13:12:58.380198 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 13:12:58.382417 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.382418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.382418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.382660 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.383077 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 13:12:58.383078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 13:12:58.383079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 13:12:58.383160 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 13:12:58.383510 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 13:12:58.383511 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 13:12:58.383576 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 13:12:58.383638 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 13:12:58.383752 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 13:12:58.384466 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:58.368320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.8.88.188:38185)
0928 13:12:58.368386 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.368388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.368396 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:12:58.368437 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.368437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.368448 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.368494 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.368961 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.369714 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.370839 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:58.371160 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:58.371234 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:58.380294 (+ 9060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 13:12:58.380296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 13:12:58.380296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 13:12:58.380320 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 13:12:58.382292 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.382755 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.382756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.382757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.382836 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 13:12:58.383272 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 13:12:58.383273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 13:12:58.383274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 13:12:58.383408 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 13:12:58.383465 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:58.384294 (+ 829us) server_negotiation.cc:299] Negotiation successful
0928 13:12:58.384375 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 13:12:58.386387 31321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6: Bootstrap starting.
I20260928 13:12:58.387130 31321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6: Neither blocks nor log segments found. Creating new log.
I20260928 13:12:58.387458 31321 log.cc:824] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6: Log is configured to *not* fsync() on all Append() calls
I20260928 13:12:58.388408 31321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6: No bootstrap required, opened a new log
I20260928 13:12:58.391672 31321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } }
I20260928 13:12:58.391925 31321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 760325b0f90f4a6c870a56835dcca3f6, State: Initialized, Role: FOLLOWER
I20260928 13:12:58.392052 31321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } }
I20260928 13:12:58.392467 31445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } } }
I20260928 13:12:58.392700 31445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:12:58.392668 31294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 13:12:58.393783 31294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6 [term 1 FOLLOWER]: Refusing update from remote peer 20f83589d40b4ff0874f6b0bce185392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 13:12:58.392592 31295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "20f83589d40b4ff0874f6b0bce185392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "760325b0f90f4a6c870a56835dcca3f6"
I20260928 13:12:58.394182 31416 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 13:12:58.394719 31295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20f83589d40b4ff0874f6b0bce185392 in term 1.
I20260928 13:12:58.392575 31321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 13:12:58.396124 31445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20f83589d40b4ff0874f6b0bce185392. Latest consensus state: current_term: 1 leader_uuid: "20f83589d40b4ff0874f6b0bce185392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } } }
I20260928 13:12:58.396298 31445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 13:12:58.396766 31445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "20f83589d40b4ff0874f6b0bce185392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } } }
I20260928 13:12:58.396910 31445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 13:12:58.400504 31459 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 13:12:58.400573 31459 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 13:12:58.401201 31447 mvcc.cc:205] Tried to move back new op lower bound from 7334302426650382336 to 7334302426460585984. Current Snapshot: MvccSnapshot[applied={T|T < 7334302426474455040}]
W20260928 13:12:58.426255 31249 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:58.424487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37314 (local address 127.8.88.190:46023)
0928 13:12:58.424535 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.424537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.425552 (+ 1015us) server_negotiation.cc:407] Connection header received
0928 13:12:58.425653 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.425655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.425667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.425696 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.426164 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37314: BlockingRecv error: recv got EOF from 127.0.0.1:37314 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 13:12:58.427026 31400 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:58.424924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37938 (local address 127.8.88.188:38185)
0928 13:12:58.424965 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.424967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.426510 (+ 1543us) server_negotiation.cc:407] Connection header received
0928 13:12:58.426604 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.426605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.426617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.426646 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.426963 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37938: BlockingRecv error: recv got EOF from 127.0.0.1:37938 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 13:12:58.427784 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:58.425284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.8.88.189:33851)
0928 13:12:58.425323 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.425325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.427269 (+ 1944us) server_negotiation.cc:407] Connection header received
0928 13:12:58.427370 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.427371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.427392 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.427422 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.427725 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59836: BlockingRecv error: recv got EOF from 127.0.0.1:59836 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 13:12:58.430238 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:58.429459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.8.88.189:33851)
0928 13:12:58.429495 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.429497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.429725 (+ 228us) server_negotiation.cc:407] Connection header received
0928 13:12:58.429829 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.429830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.429848 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.429875 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.430180 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59838: BlockingRecv error: recv got EOF from 127.0.0.1:59838 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 13:12:58.433434 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:58.432572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.8.88.189:33851)
0928 13:12:58.432635 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.432637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.432899 (+ 262us) server_negotiation.cc:407] Connection header received
0928 13:12:58.433003 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.433004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.433015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.433043 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.433387 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59850: BlockingRecv error: recv got EOF from 127.0.0.1:59850 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260928 13:12:58.437408 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:58.436607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.8.88.189:33851)
0928 13:12:58.436651 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.436653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.436862 (+ 209us) server_negotiation.cc:407] Connection header received
0928 13:12:58.436994 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.436995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.437007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.437035 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.437364 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59856: BlockingRecv error: recv got EOF from 127.0.0.1:59856 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 13:12:58.443850 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:58.442772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.8.88.189:33851)
0928 13:12:58.442866 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.442868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.443160 (+ 292us) server_negotiation.cc:407] Connection header received
0928 13:12:58.443335 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.443336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.443350 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.443381 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.443791 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59872: BlockingRecv error: recv got EOF from 127.0.0.1:59872 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 13:12:58.454444 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:58.453302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.8.88.189:33851)
0928 13:12:58.453346 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.453348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.453827 (+ 479us) server_negotiation.cc:407] Connection header received
0928 13:12:58.453935 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.453936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.453948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.453979 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.454382 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59880: BlockingRecv error: recv got EOF from 127.0.0.1:59880 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 13:12:58.472954 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:58.471901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59890 (local address 127.8.88.189:33851)
0928 13:12:58.471944 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.471946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.472199 (+ 253us) server_negotiation.cc:407] Connection header received
0928 13:12:58.472416 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.472417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.472429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.472461 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.472856 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59890: BlockingRecv error: recv got EOF from 127.0.0.1:59890 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 13:12:58.507282 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:58.506364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.8.88.189:33851)
0928 13:12:58.506408 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.506410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.506660 (+ 250us) server_negotiation.cc:407] Connection header received
0928 13:12:58.506763 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.506764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.506793 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.506824 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.507217 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59904: BlockingRecv error: recv got EOF from 127.0.0.1:59904 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 13:12:58.577926 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:58.575954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.8.88.189:33851)
0928 13:12:58.576210 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.576212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.577058 (+ 846us) server_negotiation.cc:407] Connection header received
0928 13:12:58.577120 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.577121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.577136 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.577219 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.577854 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59908: BlockingRecv error: recv got EOF from 127.0.0.1:59908 (error 108)
Metrics: {"server-negotiator.queue_time_us":226}
W20260928 13:12:58.708468 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:58.707547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.8.88.189:33851)
0928 13:12:58.707679 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.707681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.707776 (+ 95us) server_negotiation.cc:407] Connection header received
0928 13:12:58.707876 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.707877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.707890 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.707953 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.708402 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59916: BlockingRecv error: recv got EOF from 127.0.0.1:59916 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 13:12:58.802069 31416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "20f83589d40b4ff0874f6b0bce185392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20f83589d40b4ff0874f6b0bce185392" member_type: VOTER last_known_addr { host: "127.8.88.190" port: 46023 } } peers { permanent_uuid: "760325b0f90f4a6c870a56835dcca3f6" member_type: VOTER last_known_addr { host: "127.8.88.189" port: 33851 } } peers { permanent_uuid: "ad09d25b31134ee3bdc65149fc0cef63" member_type: VOTER last_known_addr { host: "127.8.88.188" port: 38185 } } }
I20260928 13:12:58.802292 31416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f83589d40b4ff0874f6b0bce185392 [sys.catalog]: This master's current role is: LEADER
W20260928 13:12:58.966713 31325 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 13:12:58.965991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.8.88.189:33851)
0928 13:12:58.966066 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:58.966069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:58.966092 (+ 23us) server_negotiation.cc:407] Connection header received
0928 13:12:58.966193 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:58.966194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:58.966206 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:58.966275 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 13:12:58.966644 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59928: BlockingRecv error: recv got EOF from 127.0.0.1:59928 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 13:12:59.323947 31413 catalog_manager.cc:1259] Loaded cluster ID: 08156af25934493e9c49e711d3e9ec65
I20260928 13:12:59.324024 31413 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63: loading cluster ID for follower catalog manager: success
I20260928 13:12:59.324846 31413 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63: acquiring CA information for follower catalog manager: success
I20260928 13:12:59.325198 31413 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ad09d25b31134ee3bdc65149fc0cef63: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:12:59.401183 31459 catalog_manager.cc:1259] Loaded cluster ID: 08156af25934493e9c49e711d3e9ec65
I20260928 13:12:59.401319 31459 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6: loading cluster ID for follower catalog manager: success
I20260928 13:12:59.402164 31459 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6: acquiring CA information for follower catalog manager: success
I20260928 13:12:59.402547 31459 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 760325b0f90f4a6c870a56835dcca3f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 13:12:59.483032 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:59.479999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.8.88.189:33851)
0928 13:12:59.480179 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:59.480181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:59.480192 (+ 11us) server_negotiation.cc:407] Connection header received
0928 13:12:59.480219 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:59.480219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:59.480268 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:59.480350 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:59.480530 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:59.481271 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:59.482209 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:59.482415 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:59.482482 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:59.482705 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:59.482824 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:59.482897 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:59.482941 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 13:12:59.482963 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260928 13:12:59.488256 31535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:59.485330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.8.88.188:38185)
0928 13:12:59.485429 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:59.485431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:59.485439 (+ 8us) server_negotiation.cc:407] Connection header received
0928 13:12:59.485475 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:59.485476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:59.485512 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:59.485584 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:59.485806 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:59.486431 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:59.487447 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:59.487652 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:59.487721 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:59.487892 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:59.488009 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:59.488068 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:59.488106 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 13:12:59.488126 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":57,"threads_started":1}
I20260928 13:12:59.493330 31529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:59.490144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35638 (local address 127.8.88.189:33851)
0928 13:12:59.490225 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:59.490227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:59.490486 (+ 259us) server_negotiation.cc:407] Connection header received
0928 13:12:59.490596 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:59.490597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:59.490638 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:59.490707 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:59.490864 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:59.491427 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:59.492580 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:59.492796 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:59.492871 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:59.493064 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:59.493098 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:59.493162 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:59.493221 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 13:12:59.493244 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 13:12:59.493438 31535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:59.490185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.8.88.188:38185)
0928 13:12:59.490225 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:59.490227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:59.490468 (+ 241us) server_negotiation.cc:407] Connection header received
0928 13:12:59.490570 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:59.490571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:59.490583 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:59.490614 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:59.490806 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:59.491595 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:59.492552 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:59.492848 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:59.492934 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:59.493144 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:59.493173 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:59.493201 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:59.493313 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 13:12:59.493343 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 13:12:59.493611 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 13:12:59.490245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.8.88.190:46023)
0928 13:12:59.490381 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 13:12:59.490383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 13:12:59.490511 (+ 128us) server_negotiation.cc:407] Connection header received
0928 13:12:59.490622 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 13:12:59.490623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 13:12:59.490686 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 13:12:59.490745 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 13:12:59.490931 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:59.491690 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:59.492446 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 13:12:59.492772 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 13:12:59.492843 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 13:12:59.493066 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 13:12:59.493235 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 13:12:59.493299 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 13:12:59.493499 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 13:12:59.493535 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260928 13:12:59.495846 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 31178
I20260928 13:12:59.501084 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 31253
I20260928 13:12:59.506069 8546 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcZlb_Q/build/release/bin/kudu with pid 31328
2026-09-28T13:12:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2861 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5709 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (305054 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS